Open
Description
Elasticsearch Version
8.18.2
Installed Plugins
No response
Java Version
bundled
OS Version
cloud
Problem Description
Migrating system indices in Upgrade Assistant in kibana will result in an error due to leftover composable index templates from earlier Elasticsearch versions, which are no longer used for system indices in 9.x.
{
"features": [
{
"feature_name": "watcher",
"minimum_index_version": "7.17.29-8.0.0",
"migration_status": "ERROR",
"indices": [
{
"index": ".triggered_watches",
"version": "7.17.29-8.0.0",
"failure_cause": {
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "unable to create new index [.triggered_watches-reindexed-for-9] because it would match composable template [.triggered_watches]",
"stack_trace": "org.elasticsearch.ElasticsearchException$1: unable to create new index [.triggered_watches-reindexed-for-9] because it would match composable template [.triggered_watches]\n\tat [email protected]/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:706)\n\tat [email protected]/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:634)\n\tat org.elasticsearch.system_indices.action.GetFeatureUpgradeStatusResponse$IndexInfo.toXContent(GetFeatureUpgradeStatusResponse.java:306)\n\tat [email protected]/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993)\n\tat [email protected]/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:982)\n\tat org.elasticsearch.system_indices.action.GetFeatureUpgradeStatusResponse$FeatureUpgradeStatus.toXContent(GetFeatureUpgradeStatusResponse.java:191)\n\tat [email protected]/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993)\n\tat [email protected]/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:982)\n\tat org.elasticsearch.system_indices.action.GetFeatureUpgradeStatusResponse.toXContent(GetFeatureUpgradeStatusResponse.java:59)\n\tat [email protected]/org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:53)\n\tat [email protected]/org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:25)\n\tat [email protected]/org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:28)\n\tat [email protected]/org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:27)\n\tat [email protected]/org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:37)\n\tat [email protected]/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)\n\tat [email protected]/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)\n\tat [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)\n\tat [email protected]/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49)\n\tat [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1500)\n\tat [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1500)\n\tat [email protected]/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:434)\n\tat [email protected]/org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:383)\n\tat [email protected]/org.elasticsearch.transport.InboundHandler.executeResponseHandler(InboundHandler.java:150)\n\tat [email protected]/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:125)\n\tat [email protected]/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:98)\n\tat [email protected]/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:816)\n\tat [email protected]/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:125)\n\tat [email protected]/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:97)\n\tat [email protected]/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:62)\n\tat [email protected]/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:55)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515)\n\tat [email protected]/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378)\n\tat [email protected]/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)\n\tat [email protected]/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1447)\nCaused by: java.lang.IllegalStateException: unable to create new index [.triggered_watches-reindexed-for-9] because it would match composable template [.triggered_watches]\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:386)\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateResource(SystemIndexMigrator.java:282)\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$startFeatureMigration$7(SystemIndexMigrator.java:322)\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$updateTaskState$10(SystemIndexMigrator.java:341)\n\tat org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:257)\n\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)\n\tat org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)\n\tat org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:184)\n\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:276)\n\tat org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.lambda$execute$0(MasterService.java:576)\n\tat org.elasticsearch.cluster.service.MasterService$ExecutionResult.onPublishSuccess(MasterService.java:961)\n\tat org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:387)\n\tat org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:382)\n\tat org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:298)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:40)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.lang.Thread.run(Thread.java:1447)\n"
}
],
"type": "illegal_state_exception",
"reason": "unable to create new index [.triggered_watches-reindexed-for-9] because it would match composable template [.triggered_watches]",
"stack_trace": "java.lang.IllegalStateException: unable to create new index [.triggered_watches-reindexed-for-9] because it would match composable template [.triggered_watches]\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:386)\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateResource(SystemIndexMigrator.java:282)\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$startFeatureMigration$7(SystemIndexMigrator.java:322)\n\tat org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$updateTaskState$10(SystemIndexMigrator.java:341)\n\tat org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:257)\n\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)\n\tat org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)\n\tat org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:184)\n\tat org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)\n\tat org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:276)\n\tat org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.lambda$execute$0(MasterService.java:576)\n\tat org.elasticsearch.cluster.service.MasterService$ExecutionResult.onPublishSuccess(MasterService.java:961)\n\tat org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:387)\n\tat org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:382)\n\tat org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:298)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)\n\tat org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:40)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)\n\tat java.lang.Thread.run(Thread.java:1447)\n"
}
}
},
{
"index": ".watches",
"version": "7.17.29-8.0.0"
}
]
}
],
"migration_status": "ERROR"
}
Steps to Reproduce
- Create 7.17.28 deployment
- Create a watcher and make sure it runs at least once
- Migrate to 8.18.2
- Attempt to migrate system indices in kibana
Logs (if relevant)
No response