[pcommon.Map] Removal then Insert results in wrong value assigned.

[MovieStoreGuy]

Describe the bug
A clear and concise description of what the bug is.

Removing an attribute from pcommon.Map that is not the end of the internal list results in overwriting keys that it shouldn't be.
It only happens with a specific operation order to trigger this.

Steps to reproduce
If possible, provide a recipe for reproducing the error.

import (
	"fmt"

	"go.opentelemetry.io/collector/pdata/pcommon"
)

func main() {
	attrs := pcommon.NewMapFromRaw(map[string]interface{}{
		"instrumentation.version": "v0.0.0-local",
		"vendor":                  "none",
		"hostname":                "workstation-1",
	})

	fmt.Println("The map is:", attrs.AsRaw())
	v, _ := attrs.Get("instrumentation.version")
	if attrs.Remove("instrumentation.version") {
		fmt.Println("successfully removed instrumentation.version attr")
	}
	attrs.Insert("auto-instrumentation", v)

	fmt.Println("The map is:", attrs.AsRaw())
}

Here I pick a value that would be in the middle of a sorted list, remove it, validate that it was removed successfully and then insert an updated value for it.

What did you expect to see?

The final map should be shown as:

The map is: map[hostname:workstation-1 auto-instrumentation:v0.0.0-local vendor:none]

What did you see instead?
A clear and concise description of what you saw instead.

The resulting map is this:

The map is: map[auto-instrumentation:workstation-1 hostname:workstation-1 vendor:none]

What version did you use?
The mod file that I have is this:

module attrs-example

go 1.18

require go.opentelemetry.io/collector/pdata v0.54.0

require (
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang/protobuf v1.5.2 // indirect
	golang.org/x/net v0.0.0-20201021035429-f5854403a974 // indirect
	golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4 // indirect
	golang.org/x/text v0.3.3 // indirect
	google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 // indirect
	google.golang.org/grpc v1.47.0 // indirect
	google.golang.org/protobuf v1.28.0 // indirect
)

What config did you use?
N/A

Environment
N/A

Additional context
The idea I am trying to implement is a rename method, however, reversing the method order gives the desired output:

	attrs := pcommon.NewMapFromRaw(map[string]interface{}{
		"instrumentation.version": "v0.0.0-local",
		"vendor":                  "none",
		"hostname":                "workstation-1",
	})

	fmt.Println("The map is:", attrs.AsRaw())
	v, _ := attrs.Get("instrumentation.version")
	attrs.Insert("auto-instrumentation", v)
	if attrs.Remove("instrumentation.version") {
		fmt.Println("successfully removed instrumentation.version attr")
	}

	fmt.Println("The map is:", attrs.AsRaw())

[MovieStoreGuy]

Sorry if that isn't coherent, covid has done a number on me.

You might find this interesting though @bogdandrutu , @tigrannajaryan

[tigrannajaryan]

This is by design since Get returns a reference to the element in attribute slice. Remove then shifts the elements, so the reference points to a different element.

However, I understand how this can be an unexpected behavior. I am not sure how we can fix this without significant performance impact (we can clone in Get and the surprising behavior will be gone, but cloning is expensive).

@bogdandrutu thoughts? At the minimum we should document that the return value of Get becomes invalid after any modification to the slice.

[dmitryax]

I brought this Q some time ago as well. I believe, if we revert this performance improvement #2017, we will get this working. Not sure if we want to support this tho. cc @bogdandrutu

[dmitryax]

Here is the similar issue that shows even more unexpected outcome #5476

[dmitryax]

Removing from the 1.0 milestone since fixing it is not a breaking change for pdata API