Patience Lyrics
Did I count the days wrong?
Did we just go round and round
All the way to step one?
Every time the show's up
"So what you doin' with your life?"
It ain't givin' up
(Lay down low down)
Livin' life in phases
Another season changes
And still my days are shapeless
'Cause time waits for no one
I should be flying straight, no delay
'Cause time takes from everyone
Waiting for the day to come
Take us to the place
Takin' us where we would have gone
Showin' up in ages
"So what you doin' with your life?"
Call it givin' up
(Lay down low down)
Livin' life in phases
Another season changes
And still my ways are aimless, I know
We're growin' up in stages
And I'm runnin' out of patience
Growin' up
'Cause time waits for no one
I should be flying straight, no delay
'Cause time takes from everyone

It could be that this song is about Kevin feeling pressured to keep up the success of the band: to pump out new music and keep up with all the comments from his fans / the press. But to me, it is him thinking of a past love and how he can't move past it. What points me in this direction is his use of "we" and "us", plus many of Tame Impala's past songs have to do with relationships.
He doesn't believe it has been so long since the relationship ended; in his mind, it feels like yesterday. Could it really be that WE are back at square one, where they were before they dated. People ask him in private, when he is not putting on a face for society ("When the show's up"), why he hasn't moved on, has he just given up on love?
He is trying to grow; he is focusing on other stages and phases of his life outside of a relationship. However, his days still feel shapeless/aimless because he has no love. He knows he should have his life back on track ("flying straight") because time is running out, but hearing everyone tell him to "drive straight, don't be late" isn't helping.
He is just stuck "waiting here" until the day that takes him "where we would have gone", moving forward with the girl he still loves. He knows he is growing in other ways, and is now running out of patience waiting for growth in his relationships. He doesn't want to be aimless/shapeless anymore; he wants to make a concerted effort on this stage to get back to flying straight.