Omnifind crawler redirection to HTTPS problem
Archived post from 2009, recovered from my old blog. The technology described here is probably out of date. Original at the Internet Archive.
Problem description
Omnifind server is unable to access WebSphere Portal cluster neither directly by connecting to a single node nor through Caching Proxy.
Facts
Omnifind crawler is trying to access WebSphere Portal so called seed list and it either cannot connect or is doing an endless redirection loop.
Here are some findings:
-
the URL used by Omnifind server looks like http://host:port/seedlist/myserver
-
the application handling /seedlist URL is called Seedlist_Servlet
-
Seedlist_Servlet application contains one module wp.search.servlets.seedlist.war which is a web application
-
web application descriptor specifies that access to seed list resources requires SSL connection (CONFIDENTIAL) so the crawler is beeing redirected
-
SSL is handled by Caching Proxy server sitting between load balancer and HTTP server and:
- WebSphere Portal is not aware of SSL communication between user and Caching Proxy – that’s why it keeps trying to switch to SSL
- WebSphere Portal does not have SSL transport chain enabled because it is not needed – that’s why crawler is unable to connect directly to a single node, it gets redirected to a port which is not opened
Solution
The best would be to make the WebSphere Portal aware of SSL established between browsers and Caching Proxy but a quick workaround is to remove the SSL redirect:
- using WebSphere Application Server console export Seedlist_Servlet enterprise application
- open exported archive (I suggest using 7zip) it and then open wp.search.servlets.seedlist.war which is inside it
- then open WEB-INF/web.xml
- replace CONFIDENTIAL with NONE
- save the file
- let 7zip update EAR file
- using WebSphere Application Server console update Seedlist_Servlet with changed EAR file