Ellsbury began the 2006 season ranked by Baseball America as the sixth-best prospect in the Red Sox' organization. He started the season with the Wilmington Blue Rocks of the Class A-Advanced Carolina League. On July 3, he tied a Wilmington franchise record by stealing four bases in a game. After batting . 299 with 25 steals in 61 games and being named to the 2006 Carolina League All-Star team, Ellsbury was promoted to the Portland Sea Dogs of the Class AA Eastern League on July 12, 2006. Soon after his promotion to Portland, Ellsbury was named Eastern League Player of the Week for August 6–13. Ellsbury finished the season in Portland with . 308 batting average and 16 steals in 50 games. The Red Sox selected him as their Minor League Defensive Player of the Year and Baserunner of the Year. The team assigned him to the Peoria Javelinas of the Arizona Fall League (AFL), an off-season developmental league for top prospects. In the AFL, Ellsbury hit . 276 in 25 games. His defensive skills earned him a spot in the AFL Rising Stars All-Star game.