I'm a web architect who enjoys developing software and learning continuously. http://danielpecos.com twitter: @danielpecos Daniel has posted 1 posts at DZone. View Full User Profile

HTTP connection + HTTP Authenticacion + Proxy + SSL

12.09.2010
| 20810 views |
  • submit to reddit
Many times during your life as a java developer, you will face the situation of retrieving some resources using an HTTP connection. At first, it will seem easy, but probably some problems will arise such as:

  1. Needing to use an HTTP Proxy (maybe authentication would be required)
  2. Establishing an HTTP authenticated connection
  3. Connecting to a server that uses SSL self-signed certificates

I'm sure you will quickly find the Apache Commons solution: commons-httpclient. In this article I will show you some code that, I hope, will ease you the  resolution of the previous obstacles using this great API from Apache. First of all I will show you the basic code needed to establish an HTTP GET connection using commons-httpclient and then we will add continuous enhancements step by step:
GetMethod httpget = null;
String result = null;
try {
httpget = new GetMethod(url);

HttpClient client = new HttpClient();
int status = client.executeMethod(httpget);
if (status == 200) {
result = httpget.getResponseBodyAsString();
} else {
System.err.println("Error accessing to URL " + status + ": " +  httpget.getStatusLine());
}

} catch (MalformedURLException e) {
System.err.println("Malformed URL: " + e.getMessage());
} catch (IOException e) {
System.err.println("I/O problems: " + e.getMessage());
} catch (Exception e) {
System.err.println("URL not found: " + e.getMessage());
} finally {
if (httpget != null) {
httpget.releaseConnection();
}
}

1. Use an HTTP Proxy (maybe authentication would be required) Now lets add the necessary code to establish the HTTP connection using a Proxy. I will use a static method to configure proxy settings in our HttpClient instance:
private static void configureProxy(HttpClient client, HttpMethodBase method, String proxyHost, int proxyPort, String proxyUsername, String proxyPassword, String proxyNTDomain) {

client.getHostConfiguration().setProxy(proxyHost, proxyPort);

if (proxyUsername != null && proxyUsername.length() > 0) {

if (proxyNTDomain != null && proxyNTDomain.length() > 0) {
// use NT Domain authentication
NTCredentials credentials = new NTCredentials(proxyUsername, proxyPassword, proxyHost, proxyNTDomain);
HttpState state = client.getState();
state.setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), credentials);
} else {
// use plain user/password authentication
Credentials defaultcreds = new UsernamePasswordCredentials(proxyUsername, proxyPassword);
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), defaultcreds);
}
method.setDoAuthentication(true);
}

}
and then we will have to insert this line of code in our previous example (I have inserted it between lines 6-7):
configureProxy(client, httpget, proxyHost, proxyPort, proxyUsername, proxyPassword, proxyNTDomain);
At this moment, we have point 1 solved, so let's go for point 2: Basic HTTP Authentication.

2. Establish an HTTP authenticated connection In order to keep code as clean as possible, I will create another static method to set HTTP authentication:
private static void configureHTTPAuthentication(HttpClient client, String host, int port, String httpUsername, String httpPassword) {
client.getState().setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(httpUsername, httpPassword));
}
and the required call to this method, just after the previous one:
configureHTTPAuthentication(client, httpget.getURI().getHost(), httpget.getURI().getPort(), httpUsername, httpPassword);
This one was easy ;-)

3. Connect to a server that uses SSL self-signed certificates Now lets solve the last point, probably the more difficult of all of them. We will need two implementations of ProtocolSocketFactory and X509TrustManager that accept self-signed certificates: EasySSLProtocolSocketFactory and EasyX509TrustManager, which you can find in httpclient-contrib (I have grabbed them from Adrian Sutton and Oleg Kalnichevski and you can also find them attached at the end of this article). Once in our classpath, we have to insert the following code in our first example, just below the call to configureHTTPAuthentication:
if (url.startsWith("https")) {
Protocol protocol = new Protocol("https", new EasySSLProtocolSocketFactory(), 443);
client.getHostConfiguration().setHost(httpget.getURI().getHost(), 443, protocol);
// we also can set this socketfactory as global for all the connections
//Protocol.registerProtocol("https", protocol);
}
Finally, we should be able to connect to a HTTP-Authenticaded SSL URL, using an http (authenticated-?)proxy. Isn't it cool? This is how our final code looks like:
private static void configureProxy(HttpClient client, HttpMethodBase method, String proxyHost, int proxyPort, String proxyUsername, String proxyPassword, String proxyNTDomain) {

client.getHostConfiguration().setProxy(proxyHost, proxyPort);

if (proxyUsername != null && proxyUsername.length() > 0) {

if (proxyNTDomain != null && proxyNTDomain.length() > 0) {
// use NT Domain authentication
NTCredentials credentials = new NTCredentials(proxyUsername, proxyPassword, proxyHost, proxyNTDomain);
HttpState state = client.getState();
state.setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), credentials);
} else {
// use plain user/password authentication
Credentials defaultcreds = new UsernamePasswordCredentials(proxyUsername, proxyPassword);
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), defaultcreds);
}
method.setDoAuthentication(true);
}

}

private static void configureHTTPAuthentication(HttpClient client, String host, int port, String httpUsername, String httpPassword) {
client.getState().setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(httpUsername, httpPassword));
}

public static String doGETConnection(String url, String proxyHost, int proxyPort, String proxyUsername, String proxyPassword, String proxyNTDomain, String httpUsername, String httpPassword) {

GetMethod httpget = null;
String result = null;
try {
httpget = new GetMethod(url);

HttpClient client = new HttpClient();
configureProxy(client, httpget, proxyHost, proxyPort, proxyUsername, proxyPassword, proxyNTDomain);
configureHTTPAuthentication(client, httpget.getURI().getHost(), httpget.getURI().getPort(), httpUsername, httpPassword);

if (url.startsWith("https")) {
Protocol protocol = new Protocol("https", new EasySSLProtocolSocketFactory(), 443);
client.getHostConfiguration().setHost(httpget.getURI().getHost(), 443, protocol);
// we also can set this socketfactory as global for all the connections
//Protocol.registerProtocol("https", protocol);
}

int status = client.executeMethod(httpget);
if (status == 200) {
result = httpget.getResponseBodyAsString();
} else {
System.err.println("Error accessing to URL " + status + ": " + httpget.getStatusLine());
}

} catch (MalformedURLException e) {
System.err.println("Malformed URL: " + e.getMessage());
} catch (IOException e) {
System.err.println("I/O problems: " + e.getMessage());
} catch (Exception e) {
System.err.println("URL not found: " + e.getMessage());
} finally {
if (httpget != null) {
httpget.releaseConnection();
}
}

return result;

}

Possible exceptions But, if we haven't been lucky, we can find one of this two exceptions (if not both) when trying to establish connection:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This one is due to the JDK can not verify the self-signed certificate. We have to add it to a trusted keystore and make it available to the JDK, using InstallCert java standalone application (more info here: http://blogs.sun.com/andreas/entry/no_more_unable_to_find) Another different problem that I got while developing this code, was the next one:

javax.net.ssl.SSLKeyException: RSA premaster secret error

It's related to this size of the certificate and the JDK cryptographic capabilities. To resolve this issue, download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files version 5.0. Extracted from its README file:

Due to import control restrictions, the version of JCE policy files that are bundled in the JDK(TM) 5.0 environment allow "strong" but limited cryptography to be used. This download bundle (the one including this README file) provides "unlimited strength" policy files which contain no restrictions on cryptographic strengths. Please note that this download file does NOT contain any encryption functionality since such functionality is supported in Sun's JDK 5.0.Thus, this installation applies only to Sun's JDK 5.0, and assumes that the JDK 5.0 is already installed.

And finally... I hope this article would help you to resolve your HTTP connection issues. If not, please post a comment and I will try to reply you ASAP. You can also find me at twitter @danielpecos Files used for this demo:

  • Full sources of the example developed here: httpclient-example.
  • Standalone java application to create a keystore: InstallCert
  • I also provide a mirror for the needed policy files in case of RSA premaster secret error.


You can find the original article here: http://blog.danielpecos.com/2010/12/http-connection-http-authenticacion-proxy-ssl/

Published at DZone with permission of its author, Daniel Pecos.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Pandu0134u Camel replied on Thu, 2011/01/20 - 8:43am

very good example

Jane Soares replied on Mon, 2011/02/21 - 2:40pm

Hi Daniel, I'm having trouble running your example under the company proxy. I'm getting a "Error accessing to URL 407: HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )" while running the httpclient-example and a "java.net.ConnectException: Connection refused: connect" when running the installCert. I've given the proper proxy host and credentials. I can't figure out what i'm doing wrong... For running the installCert I've setup the properties: System.setProperty("http.proxyHost","myproxy"); - exatcly as it is setup on my browser and works. System.setProperty("http.proxyPort","80"); System.setProperty("https.proxyHost","myproxy"); System.setProperty("https.proxyPort","80"); java installCert login.yahoo.com:443 I am able to connect with success to the site using the code: URL url = new URL("https://login.yahoo.com"); InputStream is = url.openStream(); Which also goes through the proxy. So i dont understand why the Socket doesnt work... Do you have any idea? If so please let me know It would be of great help :) Thank you very much...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.