Issues after upgrading ELM to 7.1

[sergiomarsen]

After upgrading doors to Version 7.1.0 I'm having some issues when performing oslcqueries with elmclient.

For instance, with previous versions I run this and worked pretty fine:

oslcquery -J XXXX -U XXXX -P XXXX -A rm,gc,jts:jts23 -p "XXXX " -E "XXXX " -G "XXXX " -q "rdm_types:ArtifactFormat = jazz_rm:Module" -O data/modules/modulesList.csv -s "dcterms:identifier,oslc:instanceShape,rdm_types:ArtifactFormat,dcterms:title" --pagesize 0

But now with 7.1 instead, it only works if I remove the -q query or produces this error:

Version 0.26.2

querying for gc config XXXX V6.0

Loading DN shapes: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 93/93 [00:31<00:00,  2.96 results/s]

Preparing Query

Exception on executing request. URL: https://xx.xx.com:1234/rm/views?oslc.prefix=rdm_types%3D%3Chttp%3A//www.ibm.com/xmlns/rdm/types/%3E%2Cdcterms%3D%3Chttp%3A//purl.o

rg/dc/terms/%3E%2Coslc%3D%3Chttp%3A//open-services.net/ns/core%23%3E%2Crm_nav%3D%3Chttp%3A//jazz.net/ns/rm/navigation%23%3E&oslc.query=true&oslc.select=dcterms%3Aidentifier%2Coslc%3A

instanceShape%2Crdm_types%3AArtifactFormat%2Cdcterms%3Atitle%2Crm_nav%3Aparent&oslc.where=rdm_types%3AArtifactFormat%3D%3Chttp%3A//jazz.net/ns/rm%23Module%3E&projectURL=https%3A//XX.XX.com%3A1234/rm/process/project-areas/_FbfG0Eu2Ee2zosH4Tf7GaA, 400, <rdf:RDF

    xmlns:rdf=http://www.w3.org/1999/02/22-rdf-syntax-ns#

    xmlns:err=http://jazz.net/xmlns/prod/jazz/foundation/1.0/>

  <rdf:Description>

    <err:detailedMessage rdf:datatype=http://www.w3.org/2001/XMLSchema#string

    >For input string: http://jazz.net/ns/rm#Module</err:detailedMessage>

    <err:errorMessage rdf:datatype=http://www.w3.org/2001/XMLSchema#string

    >Bad Request</err:errorMessage>

    <err:errorStatus rdf:datatype=http://www.w3.org/2001/XMLSchema#long

    >400</err:errorStatus>

  </rdf:Description>

</rdf:RDF>

 

Traceback (most recent call last):

  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\Scripts\oslcquery.exe\__main__.py", line 7, in <module>

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\examples\oslcquery.py", line 814, in main

    do_oslc_query(sys.argv[1:])

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\examples\oslcquery.py", line 575, in do_oslc_query  

    results = queryon.do_complex_query( args.resourcetype, querystring=args.query, searchterms=args.searchterms, select=args.select, isnulls=args.null, isnotnulls=args.value

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\oslcqueryapi.py", line 168, in do_complex_query     

    resultstack = self._evaluate_steps(querycapabilityuri,querysteps, select=parsedselect, prefixes=prefixes

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\oslcqueryapi.py", line 341, in _evaluate_steps      

    results = self.execute_oslc_query(querycapabilityuri,whereterms=[step], select=select, prefixes=prefixes, orderbys=orderbys, searchterms=searchterms, show_progress=show_progress,

maxresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=pagesize, verbose=verbose, saverawresults=saverawresults, cacheable=cacheable, intent="Perform OSLC Query")    

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\oslcqueryapi.py", line 438, in execute_oslc_query   

    results = self._execute_vanilla_oslc_query(querycapabilityuri,query_params1, select=select, prefixes=prefixes, show_progress=show_progress, verbose=verbose, maxresults=maxresults

, delaybetweenpages=delaybetweenpages, pagesize=pagesize, intent=intent, saverawresults=saverawresults, cacheable=cacheable)

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\oslcqueryapi.py", line 613, in _execute_vanilla_oslc_

query

    this_result_xml = self.execute_get_rdf_xml(query_url, params=params, headers=headers, cacheable=cacheable, intent=intent)

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 151, in execute_xresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=xresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=pagesize, intent=intent, saverawresults=saverawresults, cacheable=cacheable)

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\oslcqueryapi.py", line 613, in _execute_vanilla_oslc_query

    this_result_xml = self.execute_get_rdf_xml(query_url, params=params, headers=headers, cacheable=cacheable, intent=intent)

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 151, in execute_get_rdf_xml

    response = request.execute( **kwargs )

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 334, in execute

    return self._execute_request( no_error_log=no_error_log, close=close, **kwargs )

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 343, in _execute_request

    result = self._execute_one_request_with_login( no_error_log=no_error_log, close=close, **kwargs)

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 531, in _execute_one_request_with_login

    response.raise_for_status()

  File "C:\Users\XXX\XXX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\requests\models.py", line 1024, in raise_for_status

    raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://xx.xx.com:1234/rm/views?oslc.prefix=rdm_types%3D%3Chttp%3A//www.ibm.com/xmlns/rdm/types/%3E%2Cdcterms%3D%3Chttp%3A//purl.org/dc/terms/%3E

%2Coslc%3D%3Chttp%3A//open-services.net/ns/core%23%3E%2Crm_nav%3D%3Chttp%3A//jazz.net/ns/rm/navigation%23%3E&oslc.query=true&oslc.select=dcterms%3Aidentifier%2Coslc%3AinstanceShape%2Crdm_types%3AArtifactFormat%2Cdcterms%3Atitle%2C

rm_nav%3Aparent&oslc.where=rdm_types%3AArtifactFormat%3D%3Chttp%3A//jazz.net/ns/rm%23Module%3E&projectURL=https%3A//XX.XX.com%3A1234/rm/process/project-areas/_FbfG0Eu2Ee2zosH4Tf7GaA&oslc_config.context=https%3A%2F%2

FXX.XX.com%3A1234%2Fgc%2Fconfiguration%2F250

And similar happens with other queries like:

oslcquery -J https://xx.xx.com:1234/ -U XXXX -P XXXX  -A rm -p "XXXX  " -C "XXXX  " -F https://xx.xx.com:1234/rm/cm/baseline/_OG4w4FPgEe-io-vrLsIPgg -s * -O data/XXXX/reqs.csv --typesystemreport data/XXX/reqs.html -W --pagesize 0

getting below error:

Version 0.26.2

Traceback (most recent call last):

  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\Scripts\oslcquery.exe\__main__.py", line 7, in <module>

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\examples\oslcquery.py", line 814, in main

    do_oslc_query(sys.argv[1:])

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\examples\oslcquery.py", line 359, in do_oslc_query  

    c = p.find_local_component(args.component)

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\_rm.py", line 724, in find_local_component

    self.load_components_and_configurations()

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\_rm.py", line 356, in load_components_and_configurati

ons

    components_xml = self.execute_get_rdf_xml(components_uri, intent="Retrieve project's components service provider definition")

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 151, in execute_get_rdf_xml       

    response = request.execute( **kwargs )

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 334, in execute

    return self._execute_request( no_error_log=no_error_log, close=close, **kwargs )

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 343, in _execute_request

    result = self._execute_one_request_with_login( no_error_log=no_error_log, close=close, **kwargs)

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\elmclient\httpops.py", line 531, in _execute_one_request_with_l

ogin

    response.raise_for_status()

  File "C:\Users\XX\XX\Documents\PYTHON\pythonProject_test\virt_env\lib\site-packages\requests\models.py", line 1024, in raise_for_status

    raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://xx.xx.com:1234/rm/oslc_config/components

But actually, for this last case if I input https://XX.XX.com:1234/rm/oslc_config/components in my browser it downloads an *.rdf file.

Not sure is there is any change in the oslc api, encodings, authentication methods or what could happen.

I'm trying to find a workaround but if anyone has any clue would help a lot.

Thanks in advance!

[barny]

Hi

#1 - oslc.where=rdm_types:ArtifactFormat = jazz_rm:Module

I guess you upgraded from 7.0.2 on a fairly early iFix, pre-017? The rdm_types:ArtifactFormat = jazz_rm:Module certainly worked pre-017.

From iFix017 the query to check for modules is oslc.where=rdf:type=jazz_rm:Module - apparently this should have been the test for a module before 017, when I believe both styles of query where accepted, but the rdm_types query syntax isn't accepted now.

#2 I'll look at this

[barny]

#2 - not sure how I broke using the -F flag for oslcquery, but it works now in 0.27.0 and 0.27.1 - tested with both a local configuration name and a URL like you're trying to use.

[sergiomarsen]

Hi

Thanks for your help!

#1 - That worked pretty fine, just changed the query and worked with same results as before. Thanks!

About

#2 - I noticed that the issue seems is coming from requesting to: 'baseurl:9443/rm/oslc_config/components'

I have added this line to test if any behaviour changed and at least the execution is not blocked with an error:

if request.url == 'https://baseurl:9443/rm/oslc_config/components':

    remove_headers = ['X-Requested-With', 'Referer', 'OSLC-Core-Version', 'Accept', 'net.jazz.jfs.owning-context']

But of course, not sure at all what effect it may produce, just hardcoded it for testing.

Also I read herethis: "In DOORS Next 7.1 or later, a new OSLC configuration query capability is provided under the following service provider URL, https://:/rm/oslc_config/components" Not sure if this might have any relation

And just one other thing which I'm not sure if is related with this update, but before wasn't happening to me. After running a similar oslcquery as in #2, before I did get a csv with several attributes in first row, included one called 'Satisfies'. Now in contrast the execution stops throwing error like:

No type for uri='https://baseurl:9443/rm/types/LT_ghpd0RacEe-Cy7-lGVewCA'

Which if I hardcode it and don't rise the exception raise Exception( f"No type for {uri=}" ) and store it as STRANGE TYPE {uri}, the contents in csv for that strange type actually matches with contents that 'Satisfies' attribute had before. (Satisfies do not appear now in csv)

Which looks strange as in the typesystemreport the "Satisfies" is appearing with same uri as always, but the failing uri is not appearing anywhere in the document. Do you think this LT_.... id means something?

Thanks!

[sergiomarsen]

#2 - not sure how I broke using the -F flag for oslcquery, but it works now in 0.27.0 and 0.27.1 - tested with both a local configuration name and a URL like you're trying to use.

I've just tried updating also to this version with same python version you use according to readme but still have same issue

Also as a note: at line 706 of oslcquery.py python complains of nesting with same quotes, at least for me :)

Thanks!

[barny]

Ah yes I see line 706 should have '' instead of "". I'll add the -B option to my regression testing somewhere so those lines of code get covered in future releases.

I'm currently using Python 3.12.0 - not aware of relying on newer version of Python though, AFAIK elmclient should work back to 3.9 although haven't tested that.

Does the -F option work with a named configuration rather than the config URL?

Yes there's a new configuration query capability in 7.1, it's the first application-wide Query Capability for DOORS Next, which is why I haven't implemented it yet, it's in a different place than the other application-wide query capability like for GCM and ETM so I have to code for rm differently to find it. The new QueryCapability is in the components content, but it's retrieving that which is giving you the 404.

https://xx.xx.com:1234/rm/oslc_config/components is part of the discovery chain. It returns some RDF with all the projects - oslcquery will find the project name you specified with -p and then go looking for its components and configurations.

It's a protected resource but you have to be authenticated to get everything after the rootservices document.

A UUID starting LT_ is a link type, I guess that's your Satisfies link, but not sure why it won't resolve to "Satisfies" - you could try using the -Q option to disable converting URIs to helpful names, see if that works? Are you using the OOTB Satisfaction link type? Do you have a URI on your Satisfies link definition?

Odd that removing headers sort of fixes things, but those headers are pretty much on every request so why would that affect just the GET on components? Does your network have a 'protective' security proxy that you're trying to traverse to get to DOORS Next, which might for some security reason object to that request (or the response) and return 404 to elmclient, meaning the request doesn't even get to DN, or the DN response is thrown away?

[barny]

If you reapply the headers one by one, which one breaks things? I'd start by putting OSLC-Core-Version and Accept back first, then the others.

[sergiomarsen]

Hi,

Thanks for your replies!

I was using Python 3.9 and works great, now was testing with 3.11 and works same for me.

I was checking according to your information, and actually I reached some good temporary fixes.

Related with the headers, yes you are right, it's so odd it only breaks with that request, while with the others there's no problem apparently.

I checked this morning and actually with just removing 'net.jazz.jfs.owning-context' and keeping the others it does not fails, looks like that's the header which is breaking it. To be honest about the network 'protection', I'm not so sure about how our company firewall configurations could affect these requests, but there is no proxies configured afaik to reach doors instance. The other requests works fine so not sure about this.

And then, about the LT_ issue:

For clarifying, yes, the satisfaction link is added, I think in most of the projects, with the same URI defined as _baseurl_/links/satisfaction

I was digging into the xml received and after adding or rdfxml.xmlrdf_get_resource_text(resource_xml,f'.//dng_types:LinkType/rdfs:label') at the end of this line in 'type_name_from_uri' at '_rm.py' file:

id = rdfxml.xmlrdf_get_resource_text(resource_xml,".//rdf:Property/rdfs:label") or rdfxml.xmlrdf_get_resource_text(resource_xml,".//oslc:ResourceShape/dcterms:title") or rdfxml.xmlrdf_get_resource_text(resource_xml,f'.//rdf:Description[@rdf:about="{uri}"]/rdfs:label') **or rdfxml.xmlrdf_get_resource_text(resource_xml,f'.//dng_types:LinkType/rdfs:label')**

Now it resolves the name, not sure if this may cause any interference with other uris or might be a good way to get it, but at least it works to me for now.

I'm wondering if this satisfaction link is somewhere in our instance projects defined with same name but different or no uri, but... not sure why this happens now and not before while the configurations have not changed, we did just upgraded doors.

Thanks a lot for the support!

[barny]

Hi

1. net.jazz.jfs.owning-context

I think this header is a hangover from 6.x when configuration management isn't used. It may be necessary for EWM, so I've removed it just for rm, and my regression test works fine so that'll be incorporated in the next release.

Still don't understand, though, why this was a problem for you.

2. Satisfaction

I don't understand why the change you've suggested is needed, but I've also incorporated it and it'll be in the next release.

satisfaction link is somewhere in our instance projects defined with same name but different or no uri

In DOORS Next the types are local to the configuration you're querying, so while querying in a local config it doesn't matter what other definitions there are in other configurations. Querying in a GC is different, that could return types used in any of the contributions - this is not really in scope for oslcquery, i.e. I didn't think of it when I started and I still don't have a tactic to properly address it - basically oslcquery is relying on a well-built typeystem where the same type names have the same definition. See https://jazz.net/library/article/92352

HTH
Ian