"David Robert Jones", played by Jared Harris, is David Bowie's birth name. It's the second Fringe character name related to Bowie, after "Thomas Jerome Newton", played by Sebastian Roché, which is the name of the character Bowie played.
Walter says, "I asked for a sign of forgiveness, but I got nothing." He is talking about a "white tulip", which (when Peter was in the Blue Universe) Walter did in fact receive - as a drawing of a white tulip due to the events from the episode.