After a month of driving, the family gets to Welch, West Virginia. Rex's mother, Erma, refuses to be called 'Gramma'. Jeannette also meets Grandpa Walls, and Uncle Stanley; she finds it shocking that she is unable to find any resemblance between them and her father. Welch, an old coal-mining town, is the perfect portrayal of poverty and desperation, though it once flourished. Rex shows the family around and tells them that Welsh is the first place to give out food stamps. Rose Mary sees the destitution of the town as yet another opportunity to establish herself as an artist particularly since she'd have no competition. She remains focused on herself. The rest of the family feels a terrible sense of foreboding.

Brian and Jeannette are placed in special classes for children with learning disabilities at Welch Elementary School because the principal is unable to understand their accents and they are not able to understand his. The students and teachers at Welch feel that Jeannette thinks she is better than them. Eventually, Jeannette is jumped by a group of girls led by a bully named Dinitia Hewitt.

Jeannette aids an African American boy escape a dog attack one day. She helps him home and notices Dinitia watching her from across the street. Dinitia and Jeannette quickly become friends after that. Erma and Stanley, two racists, warn her that people will view her as a "nigger lover." This is a pivotal point in the memoir because Jeannette is forced to face racism as well as her mother's hypocrisy. Her parents taught her to stand up for others, and herself, and to speak her mind, especially when it concerns issues of ethics and morality. Naturally, Jeannette responds boldly and passionately to Erma and Stanley's ignorance and hate. Jeannette is very disappointed when Rose Mary suggests being polite would be better. Erma is, after all, the only one keeping them from homelessness.

Jeannette and her siblings are forced to stay with their grandparents while Rex and Rose Mary return to Phoenix to bring back some of their things and to check on the house. Erma tries to force herself on Brian, and a fight begins when Lori and Jeannette try to get Erma to stop. They are then forced to stay in the basement together. This certainly shows the reader the solidarity between the siblings. They are consistently loyal to one another. When Rex hears what happened his anger is directed at the children. He acts indifferent to what happened to Brian. This behavior makes Jeannette wonder whether he was also abused as a child. Rex has not been able to acknowledge whatever trauma he survived in childhood. His comment that Brian should have handled it like man suggests that Rex's sense masculinity and his overwhelming need to be self-sufficient may have stemmed from his family demons. Erma and Ted kick them out and they move into an old, shabby apartment that has no indoor plumbing.

Everyone except Jeannette thinks the house is irreparable. Brian and Jeannette begin to dig the foundation for the Glass Castle with Rex's encouragement. Ironically, the hole they dig is used to dump their trash, since they cannot pay the garbage bill. The dream of the Glass Castle has literally become garbage. The trash attracts rats and one night Maureen feels a rat crawling near her in her sleep. The neighborhood they live in is one of the most impoverished in Welch, and the Walls are the poorest family on their street. Despite this fact, The Walls refuse to use food stamps or church clothing drives. Jeannette befriends Kathy Pastor whose mother, Ginnie Sue, is the town prostitute. She feels they have it worse than her family.

Jeannette and Brian are teased by, Ernie Goad, a kid in the neighborhood. Ernie calls their family garbage and throws rocks at Brian and Jeannette. The siblings retaliate by devising a mattress slingshot and fling a pile of rocks on them from the roof. Rex continues to drink heavily, and one night he returns with a gash on his face and forearm. Jeannette tries to sew it up at his request. She is losing faith in her father because he continues to disappear, abandoning his family.

That winter is particularly challenging because the family cannot afford coal to heat the house. In an attempt to start a fire with kerosene, Lori accidentally gets burned continuing the fire motif. The kids argue about sleeping with the pet dogs for warmth. The cold at least masks their body odor and smelly clothing; they have no running water. One night the children realize their mother is hiding food from them. They find a large chocolate bar that she had no intention of sharing. Her children are starving, yet Rose Mary can only think of herself. So goes as far as to say that she has a sugar addiction, much like Rex's alcoholism. Mom and Dad's increasing selfishness foreshadows that their poor choices will tear the family apart.

Erma Wells dies that winter. Rex is upset by his family's lack of compassion for his mother's death and he disappears for days. Shortly after her death, Uncle Stanley burns the house down when he falls asleep smoking a cigarette. He and Ted survive and move into a two-room apartment. The Walls did occasionally visit their apartment to use the running water. Jeannette soon stops going because Uncle Stanley touches her inappropriately. Rose Mary tells Jeannette to calm down and says sexual assault is a "crime of perception."

Jeannette and Brian find a two-carat diamond ring in their yard. They ask Rose Mary to sell it so they can use the money for food and other needs, but their mother insists on keeping the ring as a replacement for her wedding ring. She feels it will boost her self-esteem; meanwhile, she ignores her children's basic needs. Jeannette tells her mother that she needs to leave Rex so that the family can apply for welfare. Rose Mary refuses to leave and scolds Jeannette for losing faith in her father. Jeannette then says she should at least get another teaching job to help the family. Rose Mary regrets her choice to have children. She blames them for her failure as an artist, and only gets a job after a child-welfare officer visits the house. Jeannette is the only one home when he visits. She dodges his questions and tells him to come back when her parents are home. She fears that the child-welfare officer will split up her siblings.

Rose Mary lands a job at Davy Elementary School just outside of Welch, but this does not alleviate their money problems. The money runs out every month and the children are once again forced to sift through garbage for food or go without. Jeannette begins seventh grade at Welch High School that fall, and she spends some time with Dinitia. She notices a change in Dinitia and she finds out later that her friend is pregnant. Jeannette also learns that Dinitia was sent to jail for stabbing her mother's boyfriend, who, Jeannette guesses, is the father of Dinitia's child.

That school year Jeannette begins working for the school newspaper, The Maroon Wave. Her faculty adviser, Miss Jeannette Bivens, was Rex's old English teacher. At the newspaper Jeannette is exposed to news stories from around the country, which allows her to grow intellectually. With knowledge comes the means of empowerment. It is at this point that she begins to question her parents' decisions, because she is learning to see the world through her own eyes. By considering news reports and comparing them to her parents' worldview, Jeannette begins to see how skewed her parents' perception of the world is. These events foreshadow the division that will occur between her parents and herself as she becomes increasingly independent with age.

Ironically, Jeannette feels sorry for the way Maureen was raised. She is turning seven, and Jeannette convinces Brian and Lori to save money for a birthday gift. Maureen spends a lot of time away from home to be with her friends' Pentecostal parents. They disagree with the Walls' parenting choices and Maureen practices their beliefs, calling herself "born-again." That summer, Rose Mary leaves to renew her teaching certification and Lori attends a government-run summer enrichment program. Jeannette is left in charge and she is determined to budget the money and feed Maureen and Brian. It's not long before Rex begs her for money, and she loans it to him every time. To pay her back, Rex takes Jeannette to a bar where he uses her as a distraction to beat Robbie in pool. Robbie drags her up to his apartment and she barely escapes. Jeannette feels her father used her. When she confronts him he blows her off saying that she was not really in any harm. He equates it to the time he threw her into the Hot Pot; she thought she would sink but he knew she could swim. With this experience she gains strength and gets her first job in a jewelry store. She is beginning to understand what it means to be a strong woman.

When Lori and Rose Mary come home at the end of the summer they appear different. Lori is bent on getting away from her family after feeling what it is like to be treated like a normal kid. After all the weeks she spent away, Rose Mary decides that she won't teach again but will pick up where she left off as an artist. Jeannette is openly belligerent to her mother, telling her that she cannot quit teaching because they need the money. Instead of facing her daughter, Rose Mary asks Rex to discipline her. Jeannette then tells him that he has not lived up to his parental duties either and Rex whips her with his belt. This pushes Jeannette to save money and get out of Welsh. By securing the one thing her parents can't - a steady job - Jeannette gains the independence and strength she needs to move forward with her life.

Two filmmakers arrive in Welch as part of a government cultural enrichment program. The filmmakers, Ken Fink and Bob Gross, are adamant that Lori will have better chances of becoming a successful artist if she moves to Manhattan. After hearing this, Jeannette tells Lori about her escape fund, and the sisters decide that Lori will move to New York City after she graduates from high school. Jeannette will follow in a year. They pool their money with Brian in a piggy bank that they name Oz. Through allusions to Oz and the Emerald City, Walls magnifies the sisters' hopes and fears about a new life in NYC (The Emerald City). That winter, Rex arrives at home with a gold Cadillac Coupe Deville named Elvis. He won the car in a game of poker.

The kids discover that Rex has stolen all of the money from Oz. After this incident, which he never confesses to, Lori stops talking to Rex. A family Jeannette babysits for offers her a job in Iowa for the summer. Jeannette asks if Lori can take the babysitting job instead so she could receive a bus ticket to New York after the summer ends. Lori's new life is beginning.

Jeannette is promoted to editor-in-chief of the school newspaper by her junior year of high school. When Chuck Yeager, a famous pilot, visits Welch High School she gets to interview him. Her role in the newspaper also gives her a social boost; she is not made fun of as she once was. Jeannette's decision to leave Welsh was based on her academic needs. Her father tries to talk her into staying, and promises to work on the Glass Castle. Her mother is simply jealous that she is not the one leaving. Rex walks Jeannette to the bus station to see her off and he gives her his favorite jackknife for protection. Mom refuses to see Jeannette off by calling goodbyes "sentimental." Walls shows that Dad no longer sees Jeannette as his cheerleader- he sees the good parts of himself in her. She is leaving Welch as he once did and, instead of feeling jealous as her mother does, he seems to want her to fair better than he did.