Debugging a Java Application with a Startup Bug

How can you remotely debug a JVM with an initialization erorr in your application?

The normal debug JVM parameters require you to attach really quickly:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -Xmx512m

By making suspend=y,timeout=5000 the JVM waits 5 seconds for an attach.
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=5005,suspend=y,timeout=5000 -Xmx512m

This made it easier to get to my IDE to attach the debugger and still have the initialization code breakpoints get triggered.

Learning something new about JUnit still!

I just discovered the ability to provide a reason for the @Ignore tag.

@Ignore(“Give a short explanation”).

This prints out wonderfully in output:

Test ‘$DayJobTest.shouldDetectProtocol8′ ignored (Depends upon currently unsupported Web Socket protocol.)

Still learning new things about JUnit. Wow.

So you want to buy a short-sale property? A California experience.

My family bought a short-sale property last year. What an incredibly rewarding experience – plenty of ups and downs. We waited 7 months for the deal to go through, and in the end we got a great deal. Since then, the market has fallen a bit further, but we are in our long-term dream home for a song!

If you can find the right property, and aren’t fazed by bureaucracy gone mad (why did Bank of America decide to lose $100,000 by waiting 6 months?) or are buying as an investment home and are not emotionally attached it can be really worthwhile.

Before we start – it is important to acknowledge a feeling of guilt you may have at profiting from someone else’s financial distres. What really helped me was to remember that while it is sad that people have arrived at a terrible financial point, by helping them achieve a short sale you are helping them get back on their feet more quickly. This is because it is better for your credit recovery to discharge through a short sale, rather than walk away, be foreclosed upon and then years later still be held as deficient and have to pay the debt anyhow.

What is a Short-Sale and Who Are the Parties in the Transaction?

A short sale happens when a property is sold, and the previous_loan-to-saleprice ratio of the property  is larger than 100%.  That is, the sellers’ house loan is larger than the value of the house – the sellers are said to be “underwater”.

Normally in a real estate transaction, there are the buyers, sellers, buyer’s agent and seller’s agent.  In a short sale, the contract-of-sale is an agreement between the buyers and sellers with their respective agents as usual, but the sale is subject to approval by the investment institution(s) that own the mortgage.  Many larger institutions hire a 3rd-party contrator who is called a “negotiator”.  The negotiator sits between the (buyer/seller) and the institutions, and is generally responsible for  the gathering of information from buyer and seller, and assembly of offers for presentation to the investing institution.

The difficulty in achieving a short sale is to obtain contract approvals, via negotiators, from all interested investment institutions.  This is compounded by the fact that there is no effective negotiation – many institutions take a “this it it” stance after one round of offer/counter.

In what ways does a short-sale differ from a regular sale from a buyer’s perspective?

• You really are (indirectly) dealing with the investors not sellers. (Although Seller commitment is important – see later.) The fewer investors, the better. If they are large banks expect to have long periods of inactivity followed by disappointing offers. If they are investment consortiums it may be very hard to get everyone to agree to losses of large magnitude. So as a buyer there are many ways for a deal to fall through through no fault of your own.  The offer you write needs to be attractive to the investors, NOT the sellers.
• You are subject to the resources the investors are putting in to handling short sales. At one stage, I think there were 600 short-sale contracts/deals for each staffed short-sale investigator at Bank of America – an impossible workload. You have to think about how to stand out in that crowd, so over-asking, they are never-gonna-get-a-better-deal and perhaps an injection of personal information (e.g. “my child will miss the start of school if we don’t close soon”) all help.
• It will be harder to make offers successfully on other properties if you are in contract on the short-sale property. This makes it very important to be sure you want the short sale property. We tried to compete on other houses while we waited for the short sale.  Make sure you talk with any potential buyer’s agents about this, and find out what the law says in your state.
• Sometimes mortgage and home-equity divisions at institutions are separate, so that they count as multiple investors.  For example, Bank of America have different loans and home-equity departments.  So even though it appears as if there is one institution, there are many in practice.

The Short Sale Process

This is an experience article, so the following is a summary of the short sale process my family went through.

• Make an initial offer with a reasonable 3% deposit as usual and have it accepted by the sellers. Make it contingent on the bank’s responding within a certain time.
• Paperwork submitted to bank by the sellers, who have to document that they have no money to pay the mortgage.  Make sure the sellers include this full statement of distress with the original paperwork, and as many other documents like tax returns, otherwise it will slow things down later.
• Don’t do inspections yet.
• The contract / original escrow date will likely pass by with no bank response. Chill out.
• Extend the contract timeframes by having buyers and sellers sign addendums. Weekly or bi-weekly depending on how much you love scanning/signing. Chill out.
• First bank offer arrives through selling agent. The offer will be terrible. You will counter. Consider splurging for an appraisal to help justify any counter for your own peace of mind. Chill out.
• Offer/counter back and forth at most once more. On their second offer it is usually a take it or leave it stance from the bank. Months turnaround. Chill out. Take it or leave it
• Paperwork goes to Bank #2 etc until all investors sign off. Unless there is only one loan. Chill out.
• Only get inspections done and contingencies removed after all institutions signed off, unless you need to do them earlier to remove contingencies to show you are serious.
• Proceeds as normal through escrow at the end.
• Beware that any other debtors of the sellers can make a claim against the escrow, up until the last minute. At that point, the sellers cannot pay since they are supposed to be distressed, so it will be up to you as buyer to pay. (for example, the IRS blocked the sale until some back-business-taxes were paid in our case, so I ended up paying the back-taxes through escrow to ensure the deal).
• Close and throw a party!

Where does the Stress Come From?

The stress that people talk about manifests in these ways:

• not knowing the process before you start
• when you feel like you are waiting too long with no response from sellers or institutions
• discovering there are more investment institutions than you realized that need to weigh in
• thinking that the bank is anything but a number-crunching heartless creature by personifying them or rationalizing their behavior (it’s all just numbers, baby! but is has to get to the right person’s desk first.)
• worrying the sellers may let maintenance of the property go downhill or damage the property, or remove fixtures while you all wait. (logically, there is no point worrying about something you cannot control. Write a clause and hope for the best. emotionally, this is what was hard to deal with when fixtures are removed and a pool starts going green and scaly).

How can I prepare for buying a short-sale property?

Reading the above is a good start!  Before you start, you have to find a property(!) and make sure you’ve discovered all there is to know about the agents and the sellers.  You will all be working together for some time, so this is worth it.

• Find a property. This is the hard part always .
• Research the selling agent. You are not buying the house from the sellers, but rather the institutions invested in the mortgage. However, you still need committed sellers and a great relationship with the SELLING agent who needs to be as transparent as possible, because ALL communication with the institutions goes through them.  We found out about the business dealings and partners of the selling agent, their recent track record and their relationship with the seller (how did the seller find the selling agent?).
• Research the seller.  Do they owe any back-taxes from previous businesses? Are there any debts that they are not disclosing to the institutions? This information can be hard to find, but it will save much stress if you can look at their finances/debts early, to try to determine if they are trying to hide some assets or debt. Have the sellers filed tax returns for previous years? All of these factors can slow down the deal for weeks at a time if everybody is not being straightforward. How committed to the short sale is the seller?  Will they just walk away while the banks ponder?  Why not?
• Find a buyer’s agent who has experience with short-sale buyers. I was lucky in that we had a great agent from an office with plenty of experience. Make sure that your agent has a great relationship with the selling agent. If you are in the Bay Area, ask me and I’ll give the reference of the best agent around.
• Line up a friendly mortgage broker: Work with a broker who can issue fresh approvals easily, because after 90 days where I live you need to get re-approved usually.  Keep your credit clean through the entire short-sale, and originate the loan only after all contingencies have been removed as usual.

Now that you know the people involved, it’s time to dig in to some specifics on the property, asking price and the sellers.

• Were there any prior recent listings of the home, or any recent bank offers? Oftentimes there may have been a previous short sale process that fell apart. Depending upon the reason it fell apart, you may have a large advantage and skip some steps at the bank, because they already have stated a position on what they can accept. If it fell apart because the buyer pulled out, having got a final offer from the bank, you have skipped about 2-3 months of waiting. If the deal(s) fell apart because the bank refused a buyer counter-offer, you at least have some data to work with. The easiest way to find this out is from the selling agent, if they are kind enough to share…
• How did the selling agent arrive at the asking price? How does this relate to comparable homes in the neighborhood? How many loans are current on the property, what are the values and what kinds of institutions hold the loans?The selling agent needs to carefully justify the asking price. Investors are unlikely to accept large losses. The exact formula varies apparently. It is wise to be the highest cash offer, and above asking if the comparables determine so. You might consider paying for an appraisal (I did) to work out if your offer (especially if over asking) is justifiable.
• Are there other offers on the table right now, or other interested parties? If you make an offer, will they still look at other offers with less cash?  Once you make an offer, it helps if the seller chooses to reject other offers, especially if they are for less money. Multiple offers confuses the institutions and causes delay. Ideally the selling agent simply presents one (best) offer, above asking, and at a high repayment percentage for the investors.

Conclusion

Short-sales offer fantastic potential, with many ways to fall through.  If a deal sounds too good to be true – it is.  Do the research on the asking price, get an appraisal to double-check you are not wasting money.

Make your offer appealing to the investors: a sole offer that is over asking and comparable with other recent sales seems to work well.

Case Insensistive Regular Expressions in Java

Thanks to this post, I’ve discovered how to use case insensitive regular expressions.

Simply start your regular expression with “?i:”.  So “(?i:REGEX)” matches REGEX case insensitively.

In the code example below, one can use this to extract out an attribute value inside quotes when the attribute name is case-insensitive.

public static void main(String[] args) {
        String example = "somePrefix NAME=\"value\" someSuffix";

        Pattern p = Pattern.compile("(.*)\\s(?i:name=)(\"(.*)\")\\s(.*)");
        Matcher m = p.matcher(example);
        if (m.matches()) {

            MatchResult mr = m.toMatchResult();

            for( int i = 0; i <= mr.groupCount(); i++) {
                System.out.println("Group "+i+": "+mr.group(i));
            }
        }

        if ( m.matches() && m.groupCount()>=3) {
            System.out.println("Extracted value: "+m.group(3));
        }

    }

The code output yielded is:

Group 0: somePrefix NAME="value" someSuffix
Group 1: somePrefix
Group 2: "value"
Group 3: value
Group 4: someSuffix
Extracted value: value

Dynamic URL Class Loaders: A Simple Use Case

I recently discovered the usefulness of dynamically loading classes into a JVM and thought I would document the discovery.

Java has a class called a URL class loader (URLClassLoader). You can create one using a URL and install it as the current thread’s class loader. Any references to new classes will use the URL class loader first before looking at the parent class loader.

URL[] urls = ...
ClassLoader originalClassLoader = Thread.currentThread().getClassLoader();
ClassLoader newClassLoader = new URLClassLoader(urls, originalClassLoader);

try {    
    Thread.currentThread().setContextClassLoader(newClassLoader);
    // write code to load new classes
} finally {
    Thread.currentThread().setCLassLoader(originalClassLoader);
}

The example and the way I used this was for loading in JNDI initial contexts. See this tutorial for an example.
One can use this to load groups of classes together from a local jar file.
I found it useful to extend this a little with the ability to walk a file system searching for jar files to load using a single class loader.

public class JarSeekingURLClassLoader extends URLClassLoader {

    public JarSeekingURLClassLoader(File file, ClassLoader parent) throws MalformedURLException {
        super(makeUrls(file), parent);
    }

    private static URL[] makeUrls(File file) throws MalformedURLException {
        List<URL> urls = new ArrayList<URL>();
        urls.add(file.toURI().toURL());
        File[] jarFilesAndDirs = file.listFiles(new FilenameFilter() {
            @Override
            public boolean accept(File dir, String name) {
                return dir.isDirectory() || name.endsWith(".jar");
            }
        });
        if (jarFilesAndDirs != null) {
            for (File jarOrDir : jarFilesAndDirs) {
                if (jarOrDir.isDirectory()) {
                    urls.addAll(Arrays.asList(makeUrls(jarOrDir)));
                } else {
                    urls.add(jarOrDir.toURI().toURL());
                }
            }
        }
        return urls.toArray(new URL[urls.size()]);
    }

    public JarSeekingURLClassLoader(File file) throws MalformedURLException {
        super(makeUrls(file));
    }

    public JarSeekingURLClassLoader(File file, ClassLoader parent, URLStreamHandlerFactory factory) throws MalformedURLException {
        super(makeUrls(file), parent, factory);
    }
}

One can of course use these two patterns together to arrange to dynamically load a directory full of JAR files using a single class loader.

URL[] urls = ...
ClassLoader originalClassLoader = Thread.currentThread().getClassLoader();
ClassLoader newClassLoader = new JarSeekingURLClassLoader(new File(Config.getDynamicLibraryLocation());

try {    
    Thread.currentThread().setContextClassLoader(newClassLoader);
    // write code to load new classes
    Class.forName("com.dynamic.library.class");
} finally {
    Thread.currentThread().setCLassLoader(originalClassLoader);
}

This trick is handy if you need to simplify your compile time dependencies.

This is of course getting close to building a dynamic loading system like OSGi, but sometimes small concepts are more efficient.

Software Engineer Seeks Jobs in the Silicon Valley – An Experience

I work in Silicon Valley as a software engineer with 11 years experience, and recently I started looking for a new job.  I had 5 offers in 4 weeks, and made a decision to join a mid-stage startup.  People kept saying the market was “hot” right now but I tend to think it’s just warm.  Which is sobering since the bay area technology companies are doing very well.  Maybe people are more “productive” right now. This article follows me through the experience, and is hopefully helpful to others.

A number of people have given me some advice over the last 5 weeks.  The most interesting advice seems like the most difficult – don’t stop interviewing for long stretches. This was in the context of “you should always keep your interview skills fresh, and keep learning what is asked”.  I think it’s good advice, probably best applied when you’re happy in your current job, and aren’t scrambling like mad on some project.  I’d also add blog about some interesting stuff  while you are working – it will all help with your personal “brand” and confidence ultimately. So, whether you’re doing this out of necessity, or just to keep your hand in, there is definitely an art to interviewing, and it pays to prepare.

So I started looking for a job.  And feeling pretty good, until I bombed my first interview.

The Bombed Interview

After 4 years in one position, and no interviewing in between, going in to an interview unprepared was a painful wake up call.   The first interview I bombed in a number of ways.

• I had no prepared anecdotes about features I had worked on previously.  It’s really important to be able to rattle off a few stories about stuff you’ve done.  People love stories.  You need to be able to cover different angles – most technically successful project, most business-successful project, least favorite successful project, and one when failure did happen.
• I mentioned a technology that I had no real experience with, and was then asked alot about it.  This is a bad idea.  Best to not use hot technology keywords and just say “I’ve never used it in anger, but I know what it does and when to use it”.  In this case, it was memcached – I knew what it was but had never had to use it in anger.  How would it be used for a web application, what are the tradeoffs etc.  (a few large or alot of small objects…when? type questions).  Best to take this mistake as a learning experience, go home and write some code to improve yoursef.
• I did not pay enough attention to correct syntax with the technical questions and took too long on the unimportant details.  When you are answering technical questions, it’s great to be expository and animated, sharing your thought process.  But if you do that and wander off point for too long, or mess up syntax then usually the interviewer becomes uncomfortable.  That’s bad news.

Prepare the Backdrop

After that bombed interview, I did not rush off straight away into a world of recruiters, phone interviews and on-sites.  When you do get a phone- or and on-site interview, you can be pretty sure that the hiring manager will run some reference searches online on LinkedIn, and Google your name.  That’s what I would do!  So I wanted to be sure that they would find compelling content online about me.  This meant preparing a few things:  the resume, the LinkedIn profile, my personal website and this blog.  For me, I wanted consistent information, no links to distracting content (Facebook, photos etc)  and interesting content as much as possible.

I decided to approach the problem of consistent content by writing my resume in a Word document, and then using that content for both my LinkedIn profile, and my personal website.   This way I could add details on LinkedIn and my website that added richness that I could not fit into the resume due to space limitations.

• Resume: I started by designing my resume a little differently: it was a 1 page, 2 column layout that I eventually expanded to 2 pages to include more technical detail, on the advice of a couple of recruiters along the way.  The effect of this was to purposefully disrupt the “scanning” behavior of hiring manager who scan left-to-right full page for interesting tidbits. With the 2 column layout, many said it was harder to find information –  but it also meant they paid attention a little more.  A trade-off I thought desirable – the left side was about contact details, a photo and objectives/skills, while the right side was employment history with tight text about achievements/responsibilities.  I also flipped these descriptions into third-person.  This made my resume sound more like a consultant’s profile, avoid’s the “starting too many sentences with I” problem, and reads slightly more fluently and formally to me.
• LinkedIn Profile:  I put quite a fair amount of time into my LinkedIn profile anyhow.  It pays to update it every now and then (I try annually) to get re-indexed and add new skills and publications or other new sections that are now supported.  I took the time to pick three old publications, including my Ph.D. thesis, and added some new skills.  I took the existing positions and filled in the content with that from the resume.  THe LinkedIn folks want this to BE your resume – but for better or worse there are some old-style recruiters that still need to expect a resume.  More on this below.
• Personal Website: I decided to research templates for HTML5 sites.  This way, my content would look nice on many devices that access the site.  I developed a basic “Home/Experience/Contact” site and used the resume content again.  this time, I added a narrative on the home page about what I had done previously.
• Blog: I had blogged here on a few topics while working.  I’ve linked this to my LinkedIn Profile so when I post an article, it shows up there.  I also set up a ping.fm account to advertise posted articles on Facebook, LinkedIn and Twitter all at once.

Prepare The Stage

Before engaging the real world, I found it very central to visualize explicitly what I wanted in a new job.  I took all the positives from the last job and spent time designing what my own engineering team and process would look like.  In my case, I ended up looking for a mid- to end- stage startup with a challenging, interesting domain.  My criteria ended up being:

• location / commute
• business domain
• perceived compatibility between myself,  management and future colleagues.
• technology stack and engineering process
• benefits package

Start the Show, Act I: Contacts, Recruiters and Companies

I did not really have any recruiters from previous efforts, so this search started with people I knew (always great to find companies through them)  and took the approach of going with many recruiters at once.  I became a job seeker and open link member on LinkedIn, and replied to all recruiters who had previously and were now getting in touch with me.  This means I was actively engaged in about 20 contacts at once.  Gmail Calendaring on my iPhone became vital.  I also liked that I always sounded busy because it was hard to find a slot for a phone interview.  The flip side is that managing these people and dealing with phone calls becomes time consuming.  However, I think it’s important to go wide initially – I have an idea on who the best recruiters are and what their approach is.  It sounds cliche, but I’d look for recruiters who use LinkedIn, have recommendations and spend far more time asking about you rather than just getting you to look at companies.  Also, if they do talk about companies, they spend some time talking about the hiring manager.

I found after a few days it was harder for me to source companies of interest by myself, and I’d exhausted the initial batch of recruiters.  I then started seriously looking at online job boards: stack overflow / stack exchange / linkedin / careerbuilder /monster / craiglist.   I also started looking at Y Combinator and other venture capital sites for companies that were funded 12-24 months ago that looked like they were doing well.  That’s how I found companies like Evernote and oDesk.

Act II: The Phone Interview

I had very standard and very challenging phone interviews.  One company even had me perform a  non-trivial programming task that took me 10 hours to complete.  And it wasn’t free work – it was more like a challenging all-day exam question.  I was very impressed by that, and since I eventually stumbled upon the “right” solution after much thought, it gave me a much needed confidence boost! In the end I must have done about 20 phone interviews.  I am still waiting to hear back from a few(!), some said no after a day, and some said sure.

Standard Phone Interview Questions (asked >2 times)

• Why did you leave your last job?
• What kind of job are you looking for?
• Your web request is running slowly, what could be the reasons?
• Tell me about a successful technical project?
• Have you managed reports?

Interesting Phone Questions

• When would you use an LRU cache or a LFU cache?
• How would you sort a 1TB file of integers using 1G of RAM and infinite disk space?
• How do you improve the performance of a database query?

When asked if you have any questions, I think it helps to show interest in the company but not too many details of pay, or benefits – it’s too early.  I try to focus on how they make money, what the commute is like for the interviewer, how many engineers they have, and how they do releases and testing.  That shows an interest, without getting too detailed on any one area.

Act III: The On-Site Interview

In the valley, expect 3-6 hours of interviews.  I’ve been to about 7 on-sites in my process this time, and had offers from 5.  I’ve had a number of different experiences – from meeting VPs and management initially, to meeting engineers and product folks but not the hiring manager.  Most times you will be asked to code on a whiteboard.  If not, be wary!  When coding on a whiteboard I found it is very important to be happy, alert, open and to explain your thinking without getting too chatty, or distracted from the problem at hand.

WIth technical questions,  if you’re good, you’ll get by on intuition, design skills and by reviewing your data structures textbook (linked list implementations and hash tables/caches, or other cool lesser known data structures.)  Rather than focussing on the actual technical interview questions here, I’m going to defer that to another article; rather I’d like to suggest questions to ask which turn out to be pretty critical I think in showing how you think and your interest in a company.

Questions to ask VPs and CEOs

• Describe how you split your department into groups and how it is working
• What’s one thing you would like to improve about your company/department’s work process?
• Besides hiring, what’s your biggest challenge technically in the groups right now?
• How are you approaching that?
• what are your sources of revenue?
• what’s the exit strategy and thinking on timing at the current time?

Questions to ask Managers

• How do you develop your staff professionally?
• Are people working 80 hour weeks?  How flexible are the working hours?  Can we work from home 1-2 days a week?
• What’s the biggest engineering process improvement youa re working on?
• How do you evaluate whether the process is working?
• What’s your philosophy about software testing – how do you determine the quality of the software each release?

Questions to ask Future Colleagues

• What’s the one thing you would change about your working environment?
• How are production crises managed?  Post mortems or responses – are they considered and thoughtful?
• How does the engineering department use QA (if they are separate)?
• Where’s good for lunch around here?  ( if it’s going well )

Act IV: Negotiations

Value yourself well.  You’ve done wonderful things for your last company, and you’re good at working with people and not just coding, right?

I suggest being forthcoming and aggressive when talking numbers with your hiring manager.  I think it’s a good idea to state your previous salary and what you’re looking for now. And why.

Don’t forget about reviewing glassdoor.com and payscale.com, and (with a grain of salt) asking recruiters what the salary ranges are right now.

I’d also say be firm up front.  Let them come back to you with an offer  If they come back with a lower offer, provide alternatives.  Can you play with benefits?  Can you suggest adjusting the  mix between equity/payment?

Of the 5 offers I had, I improved 2 of them by significant amounts, another offered $20k less than my aggressive number, and the others got closer (but less than) my aggressive number.  One company (perhaps wisely) wanted me to truly want to go there, so rather than made a numeric offer they effectively said “come back to us if you really want to work here”.  I just don’t think that’s really giving me enough detail after meeting with them and having a 6 hour interview!

In the end, for the style and title I wanted, because I went aggressive, I can be pretty confident I’m not getting totally ripped off in the current market.

Conclusions

I think reading this should give you some ideas on how to start looking for a job, and some things to think about.  I don’t really see it as prescriptive, but hopefully it will prod me in the future to get out there and give me a boost on what to think about when I go out there.  Ultimately, I’m really lucky that I work in a great, fun field with plenty of opportunity here in the valley.

Web Application Security – Part 2 – WebGoat Tutorials

Tonight I am trying to install WebScarab so I can complete the webgoat tutorials (at least some of them).

I have to back track and install java on my laptop to run the installer (java jdk – web scarab – web goat is the stack so far).

I’d like to work through WebGoat and get to SQL injection attacks – I’d like to be able to see what that looks like, basically because this xkcd comic is hilarious: http://xkcd.com/327/.

Ack:  webgoat died in java 6 with this exception from the installer:

Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
at com.izforge.izpack.panels.ShortcutPanel.isValidated(ShortcutPanel.jav
a:575)
at com.izforge.izpack.installer.InstallerFrame.navigateNext(InstallerFra
me.java:914)
at com.izforge.izpack.installer.InstallerFrame$NavigationHandler.actionP
erformed(InstallerFrame.java:957)

Easily solved – simply go to \Prgram Files\WebScarab and launch the webscarab.jar by hand – the only thing this stops is the shortcuts being created. So now I’m wondering about the wisdom of going with Java 6 AWT with this tool that is from 2007, but I’ll give it  a go.

Now I am running WebScarab, I decided to also install Wireshark to have around.  Already had Firebug installed too.

So I just tested and webscarab appears to be working.  It appears to be nicer than tcpmon from axis.  I’m going to try it out – editing a request on-the-fly seems like a nice feature for debugging issues.

First Attack Vector – HTTP Splitting

Okay, so I am just working through the HTTP splitting attack – this is using carriage return and line feeds to inject fake responses.

This can obviously be avoided by filtering input characters to exclude control characters.  An important concern for internationalization.

I think the key idea is to fool the browser into doing a 302 redirect with a parameter that can contain an encoded line feed and “HTTP/1.1 200 OK” header.  When the server writes out the Location: header for the redirect, we end up simulating a 200 OK on a new line.  The browser interprets this as a second response and you can inject whatever content you like – add an accurate content-length and the browser renders it and whats more stops at the end of your content.

Example:  (see also http://www.securiteam.com/securityreviews/5WP0E2KFGK.html):

someparameterval
Content-Length: 0

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 21
<html>Hacked</html>

One arranges for a 302 to happen where a parameter has a value like:

someparameterval%0d%0aContent-Length:%200%0d%0a%0d%0aHTTP/1.1%20200%20OK%0d%0aContent-Type:%20text/html%0d%0aContent-Length:%2022%0d%0a%0d%0a<html>Hacked</html>

This ends up rendering a response like:

 

HTTP/1.1 302 Found 
Date: Tue, 14 Dec 2010 11:11:07 GMT
Server: Apache/1.3.29 (Unix) mod_ssl/2.8.16 OpenSSL/0.9.7c
Location: http://attackedhost.com/someurl?parameter=someparameterval
Content-length: 0
HTTP/1.1 200 OK
Content-length: 22 
<html>Hacked</html>

…rest of 302 headers….

 

This attack vector supports others (cache poisoning for example uses a future Last-Modified header with this vector to fool browsers into caching a attack site).  In a sense this attack is not “pure” – it relies on browsers being “naive” and starting processing of the second response in the middle of the 302 response.  Less than satisfying….

One thought is that many companies protect request parameters but not response ones – we could attack from within a company by setting funny values in these parameters too on the server responses.  That would be more interesting to protect against….

 

Generate your Website Database Layer: NORM not ORM

What is NORM?

The “Natural Object Relational Model”  is a play on “Object Relational Model” (ORM).  The key idea is to completely generate your database layer from the natural relational schema first, using standard ORM patterns.  This is the opposite of what many top-down methodologies (hibernate) and model-driven “CASE” tools provide.  It allows for accurate modelling of legacy and larger relational databases also.

Normally, database layers for websites are model based.  One is supposed to start with the objects, model them in some notation, and possibly annotate them to describe how they are stored.  Then the database or interpreting engine transforms the models into database classes, whose object instances at runtime generate queries – most ORM database systems generate the DML statements on the fly based on object values and annotated “instructions”.

The idea of NORM is to generate up-front all the code from the relational database itself, using database metadata and “instructions”.  This makes it possible to work with legacy databases.

I’ll go into more detail in a future post about some of the code, and what instructions look like, but the key concept is to interpret the existing database schema, with some declarative instructions on how to generate database classes that handle natural, inner and outer joins, aggregate functions and even group by functionality.  See how we can also pass-through database-specific syntax for SQL when needed.

In a nutshell, this has worked well at a couple of startups so far – the generated classes model database table accesses, and one can support natural and outer joins for objects declaratively with the “instructions”.

Why generate a database layer?

Almost everything you need is defined in the schema already – one should not have to repeat one’s self.  With a minimal set of instructions, one can generate 100% of your database/persistence layer.  One can even use this technique for no-SQL “schema-less” databases by inferring a schema from sample data.

How do I handle complex database-specific queries?

NORM systems always allow for a pass-through that allows complex queries.  Try to avoid them.  But if you must use database-specific features for efficiency….you should be able to.

How is this different from Hibernate-type systems?

We start by modelling the database itself and generating abstract objects, rather than abstract models turning into database access.  We enjoy all the same advantages in terms of the DRY principle and naming consistency.  Additionally, generating the database layer yields benefits in that we can adorn generated classes with extra behavior (e.g. performance tracers) by changing the generator.

 

Web Application Security – Beginnings

I’m tentatively hoping to spend Monday nights working on some web security knowledge improvement.  In my day job this comes up every now and then.  The motivation for spending some time on web application security is that a while back, I worked with the folks who set up XSS protection for the day job, and we came up with a pretty cool regular expression.  Also, it seems like because I have played with the authentication mechanisms a few times for the current day job, authentication integrity and site protection from lawsuits seems to crop up a bit.

Recently I’ve found Gruyere and WebGoat to be two very interesting and easy to set up starting points.  I also want to investigate FireSheep and play with it  a little as well.  I’m going to start with WebGoat due to the wealth of quality material on the owasp.org website to provide explanations, and a lack of Python familiarity to date.

WebGoat

 

WebGoat is a Tomcat/java web application that can be downloaded via maven or a pre-configured zip file with java and tomcat preconfigured.  Despite the 60+MB I decided it would be quicker to download a self-contained application to get up and running.

While I was waiting for the download a good read is the OWASP top ten for 2010: http://www.owasp.org/index.php/Top_10_2010-Main

 

 

 

 

Netflix grabs 20 percent of peak time U.S. traffic | The Digital Home – CNET News http://ping.fm/IQ2FB