Trajectory prediction plays an important role in autonomous driving. Autonomous driving trajectory prediction refers to predicting the future driving trajectory of the vehicle by analyzing various data during the vehicle's driving process. As the core module of autonomous driving, the quality of trajectory prediction is crucial to downstream planning control. The trajectory prediction task has a rich technology stack and requires familiarity with autonomous driving dynamic/static perception, high-precision maps, lane lines, neural network architecture (CNN&GNN&Transformer) skills, etc. It is very difficult to get started! Many fans hope to get started with trajectory prediction as soon as possible and avoid pitfalls. Today I will take stock of some common problems and introductory learning methods for trajectory prediction!

Introduction related knowledge

1. Is there an entry order for the preview papers?

A: Let’s first look at the sequential network, graph neural network and Evaluation in survey, problem formulation, and deep learning-based methods.

2. Is behavior prediction trajectory prediction?

Coupling and behavior are not the same. Coupling usually refers to the actions that the target vehicle may take, such as changing lanes, parking, and overtaking. , accelerate, turn left, turn right or go straight. The trajectory refers to a specific future location point with time information.

3. Among the data components mentioned in the Argoverse data set, what do labels and targets refer to? Does labels refer to the ground truth within the time period to be predicted?

In the table on the right, the OBJECT_TYPE column usually represents the self-driving vehicle itself. The data set usually specifies one or more obstacles to be predicted for each scene, and these targets to be predicted are called targets or focal agents. Some datasets also provide semantic labels for each obstacle, such as vehicles, pedestrians, or bicycles.

Q2: Are the data forms of vehicles and pedestrians the same? I mean, for example, one point cloud point represents a pedestrian, and dozens of points represent vehicles?

A: This kind of trajectory data set actually gives the xyz coordinates of the center point of the object, both for pedestrians and vehicles.

Q3: The argo1 and argo2 data sets are only specified. A predicted obstacle, right? How to use these two data sets when doing multi-agent prediction

argo1 only specifies one obstacle, while argo2 may specify as many as twenty. However, even if only one obstacle is specified, this does not affect your model's ability to predict multiple obstacles.

4. Path planning generally considers low-speed and static obstacles. What is the role of combining trajectory prediction? ? Key snapshot?

A: "Predict" the self-vehicle trajectory as the self-vehicle planning trajectory, you can refer to uniad

5. Trajectory prediction has high requirements for the vehicle dynamics model ? Do you just need mathematics and automotive theory to establish an accurate vehicle dynamics model?

A: nn network is basically not required, rule based requires some knowledge

6. A vague novice, where should I start to expand? Knowledge (not able to write code yet)

A: First read the review and sort out the mind map, such as "Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research" Directions" for this review, please read the original English text

7. What is the relationship between prediction and decision-making? Why do I feel that prediction is not that important?

A1(stu): 默認(rèn)預(yù)測(cè)屬于感知吧，或者決策中隱含預(yù)測(cè)，反正沒(méi)有預(yù)測(cè)不行。A2(stu): 決策該規(guī)控做，有行為規(guī)劃，高級(jí)一點(diǎn)的就是做交互和博弈，有的公司會(huì)有單獨(dú)的交互博弈組

8. At present, for leading companies, do general predictions belong to the large perception module or the large regulation module?

A: Prediction is based on the trajectory of other cars, and control is based on the trajectory of the car. The two trajectories also affect each other, so prediction is generally based on regulation.

Q: Some public information, such as Xiaopeng’s perception xnet, will produce prediction trajectories at the same time. At this time, I feel that the prediction work is placed under the perception module, or that both modules have their own predictions. Modules, different goals?

A: They will affect each other, so in some places prediction and decision-making are a group. For example, if the trajectory planned by your own car is intended to squeeze other cars, other cars will generally give way. Therefore, some work will regard the planning of the own vehicle as part of the input of other vehicle models. You can refer to M2I (M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction). This article has similar ideas. You can learn about this lane of PiP: Planning-informed Trajectory Prediction for Autonomous Driving

9.argoverse How do you get the center line map at the intersection where there are no lane lines?

A: Manually marked

10. If you use trajectory prediction to write a paper, where can you get it? Can the code of this paper be used as a baseline?

A: hivt can be used as a baseline, and many people use it

11. Nowadays, trajectory prediction basically relies on maps. If you change In a new map environment, will the original model no longer apply and need to be retrained?

A: It has a certain generalization ability, and the effect is not bad without retraining.

12. For multi-modal output, choose the best trajectory Is it time to choose based on the highest probability value?

A(stu): 選擇結(jié)果最好的Q2:結(jié)果最好是根據(jù)什么來(lái)判定呢？是根據(jù)概率值大小還是根據(jù)和gt的距離A: 實(shí)際在沒(méi)有g(shù)round truth的情況下，你要取“最好”的軌跡，那只能選擇相信預(yù)測(cè)概率值最大的那條軌跡了Q3: 那有g(shù)t的情況下，選擇最好軌跡的時(shí)候，根據(jù)和gt之間的end point或者average都可以是嗎A: 嗯嗯，看指標(biāo)咋定義

Trajectory prediction basic module

1. How to use HD-Map in the Argoverse data set? Can it be combined with motion forecast as input to build a driving scene graph, heterogeneous graph How to understand?

A: It’s all covered in this course. You can refer to Chapter 2, which will also be covered in Chapter 4. The difference between heterogeneous graphs and isomorphic graphs: the types of nodes in isomorphic graphs There is only one kind of connection between one node and another node. For example, in a social network, it can be imagined that node only has one type of "people" and edge only has one type of connection "knowledge". And people either know each other or they don't. But it is also possible to segment people, likes, and tweets. Then people may be connected through acquaintance, people may be connected through likes on tweets, and people may also be connected through likes on a tweet (meta path). Here, the diverse expression of nodes and relationships between nodes requires the introduction of heterogeneous graphs. In heterogeneous graphs, there are many types of nodes. There are also many types of connection relationships (edges) between nodes, and there are even more types of combinations of these connection relationships (meta-path). The relationships between these nodes are classified into different degrees of severity, and different connection relationships are also classified into different degrees of severity.

2.A-A interaction considers which vehicles interact with the predicted vehicles?

A: You can select cars within a certain radius, or you can consider cars with K nearest neighbors. You can even come up with a more advanced heuristic neighbor screening strategy yourself, and it is even possible to let the model learn it by itself. Are the two cars coming out neighbors?

Q2: Let’s consider a certain range. Is there any principle for selecting the radius? In addition, at which time step did the selected vehicles occur?

A: It is difficult to have a standard answer to the choice of radius. This is essentially asking how much remote information the model needs when making predictions. It is a bit For the second question when choosing the size of the convolution kernel, my personal rule is that if you want to model the interaction between objects at which time, you should select neighbors based on the relative position of the object at that time

Q3: In this case, do we need to model the historical time domain? The surrounding vehicles within a certain range will also change at different time steps, or should we only consider the surrounding vehicle information at the current moment?

A: Either way, it depends on how you design the model

3. What are the flaws in the prediction part of the teacher's uniad end-to-end model?

A: Just look at it. The operation of motion former is relatively conventional. You will see similar SA and CA in many papers. Nowadays, many sota models are relatively heavy. For example, the decoder will have a cyclic refine

A2: What is done is marginal prediction rather than joint prediction; 2. Prediction and planning are done separately, without explicitly considering ego and Interactive game of surrounding agents; 3. Scene-centric representation is used, without considering symmetry, and the effect is inevitable

Q2: What is marginal prediction

A: For details, please refer to scene transformer

Q3: Regarding the third point, scene centric does not consider symmetry. How to understand it?

A: It is recommended to look at HiVT, QCNet, MTR. Of course, symmetry is important for end-to-end models. The design is not easy to do either

A2: It can be understood that the input is scene data, but in the network it will be modeled to look at the surrounding scenes with each target as the central perspective, so that you can In the forward, we get the coding of each target centered on itself, and then we can consider the interaction between these codes

4. What is agent-based? center?

A: Each agent has its own local region, and the local region is centered on this agent.

5. Are yaw and heading mixed in trajectory prediction?

A: It can be understood as the direction of the front of the car

6.What does the has_traffic_control attribute in the argument map mean?

A: Actually, I don’t know if I understand it correctly. I guess it refers to whether a certain lane is affected by traffic lights/stop signs/speed limit signs, etc.

7. What are the advantages and disadvantages of Laplace loss and huber loss for trajectory prediction? If I only predict one lane line

A: Try both, whichever one works better There are advantages. For Laplace loss to be effective, there are still some details that need to be paid attention to

Q2: Does it mean that the parameters need to be adjusted?

A: Compared with L1 loss, Laplace loss actually predicts one more scale parameter

Q3: Yes, but I don’t know what use this is if it only predicts one trajectory. It feels like redundancy. I understand it as uncertainty. I don’t know if it is correct

A：如果你從零推導(dǎo)過(guò)最小二乘法就會(huì)知道，MSE其實(shí)是假設(shè)了方差為常數(shù)的高斯分布的NLL。同理，L1 loss也是假設(shè)了方差為常數(shù)的Laplace分布的NLL。所以說(shuō)LaplaceNLL也可以理解為方差非定值的L1 loss。這個(gè)方差是模型自己預(yù)測(cè)出來(lái)的。為了使loss更低，模型會(huì)給那些擬合得不太好的樣本一個(gè)比較大的方差，而給擬合得好的樣本比較小的方差

Q4：那是不是可以理解為對(duì)于非常隨機(jī)的數(shù)據(jù)集【軌跡數(shù)據(jù)存在缺幀 抖動(dòng)】 就不太適合Laplace 因?yàn)槟Ｐ托枰M合這個(gè)方差？需要數(shù)據(jù)集質(zhì)量比較高

A：這個(gè)說(shuō)法我覺(jué)得不一定成立。從效果上來(lái)看，會(huì)鼓勵(lì)模型優(yōu)先學(xué)習(xí)比較容易擬合的樣本，再去學(xué)習(xí)難學(xué)習(xí)的樣本

Q5：還想請(qǐng)問(wèn)下這句話（Laplace loss要效果好還是有些細(xì)節(jié)要注意的）如何理解 A：主要是預(yù)測(cè)scale那里。在模型上，預(yù)測(cè)location的分支和預(yù)測(cè)scale的分支要盡量解耦，不要讓他們相互干擾。預(yù)測(cè)scale的分支要保證輸出結(jié)果>0，一般人會(huì)用exp作為激活函數(shù)保證非負(fù)，但是我發(fā)現(xiàn)用ELU +1會(huì)更好。然后其實(shí)scale的下界最好不要是0，最好讓scale>0.01或者>0.1啥的。以上都是個(gè)人看法。其實(shí)我開(kāi)源的代碼（周梓康大佬的github開(kāi)源代碼）里都有這些細(xì)節(jié)，不過(guò)可能大家不一定注意到。

給出鏈接：https://github.com/ZikangZhou/QCNet

https://github.com/ZikangZhou/HiVT

8. 有拿VAE做軌跡預(yù)測(cè)的嗎，給個(gè)鏈接!

https://github.com/L1aoXingyu/pytorch-beginner/tree/master/08-AutoEncoder

9. 請(qǐng)問(wèn)大伙一個(gè)問(wèn)題，就是Polyline到底是啥?另外說(shuō)polyline由向量Vector組成，這些Vector是相當(dāng)于節(jié)點(diǎn)嗎？

A：Polyline就是折線，折線就是一段一段的，每一段都可以看成是一段向量Q2：請(qǐng)問(wèn)這個(gè)折線段和圖神經(jīng)網(wǎng)絡(luò)的節(jié)點(diǎn)之間的邊有關(guān)系嗎？或者說(shuō)Polyline這個(gè)折現(xiàn)向量相當(dāng)于是圖神經(jīng)網(wǎng)絡(luò)當(dāng)中的節(jié)點(diǎn)還是邊呀？A：一根折線可以理解為一個(gè)節(jié)點(diǎn)。軌跡預(yù)測(cè)里面沒(méi)有明確定義的邊，邊如何定義取決于你怎么理解這個(gè)問(wèn)題。Q3: VectorNet里面有很多個(gè)子圖，每個(gè)子圖下面有很多個(gè)Polyline，把Polyline當(dāng)做向量的話，就相當(dāng)于把Polyline這個(gè)節(jié)點(diǎn)變成了向量，相當(dāng)于將節(jié)點(diǎn)進(jìn)行特征向量化對(duì)嗎?然后Polyline里面有多個(gè)Vector向量，就是相當(dāng)于是構(gòu)成這個(gè)節(jié)點(diǎn)的特征矩陣么?A: 一個(gè)地圖里有很多條polyline;一個(gè)Polyline就是一個(gè)子圖；一個(gè)polyline由很多段比較短的向量組成，每一段向量都是子圖上的一個(gè)節(jié)點(diǎn)

10. 有的論文，像multipath++對(duì)于地圖兩個(gè)點(diǎn)就作為一個(gè)單元，有的像vectornet是一條線作為一個(gè)單元，這兩種有什么區(qū)別嗎？

A: 節(jié)點(diǎn)的粒度不同，要說(shuō)效果的話那得看具體實(shí)現(xiàn)；速度的話，顯然粒度越粗效率越高Q2：從效果角度看，什么時(shí)候選用哪種有沒(méi)有什么原則？A: 沒(méi)有原則，都可以嘗試

11. Is there any way to judge the smoothness of the score? If you must do it

A: This requires you to enter a flowing input such as 0-19 and 1-20 The frames are then compared with the square of the difference in scores of the corresponding trajectories between the two frames, and statistics are enough.

Q2: What indicators does Mr. Thomas recommend? I currently use first-order derivatives and second-order derivatives. But it seems not very obvious. Most of the first-order derivatives and second-order derivatives are concentrated near 0.

A: I feel that the squared difference of the scores of the corresponding trajectories of consecutive frames is enough. For example, if you have n consecutive inputs, sum them up and divide by n. But the scene changes in real time, and the score should change suddenly when there is an interaction or when going from a non-intersection to an intersection.

12. Isn’t the trajectory in hivt scaled, like ×0.01 10? . The distribution is as close to 0 as possible. I just use some methods when I see them, and I don’t use some methods. How to define the trade-off?

A: Just standardize the data. It may be somewhat useful, but probably not much

13. Why are the category attributes of the map in HiVT added to the numerical attributes after embedding, instead of concat?

A: There is not much difference between addition and concat, but for the fusion of category embedding and numerical embedding, they are actually completely equivalent

Q2: How should we understand complete equivalence?

A: Concating the two and then passing through a linear layer is actually equivalent to embedding the value through a linear layer and embedding the category through a linear layer, and then adding the two. There is actually no point in embedding the category through a linear layer. In theory, this linear layer can be integrated with the parameters in nn.Embeddding

14. As a user, you may be more concerned about HiVT What are the minimum hardware requirements for actual deployment?

A: I don’t know, but according to the information I learned, I don’t know whether NV or which car manufacturer uses HiVT to predict pedestrians, so the actual deployment is definitely feasible

15. Is there anything special about predictions based on occupation network? Do you have any paper recommendations?

A: Among the current future prediction solutions based on occupation, the most promising one should be this one: https://arxiv.org/abs/2308.01471

16. Are there any recommended papers considering prediction of planning trajectories? Is it to consider the planned trajectory of the own vehicle when predicting other obstacles?

A: This potentially public data set is difficult and generally does not provide the planned trajectory of your own vehicle. In ancient times, there was an article called PiP, Hong Kong Ke Haoran Song. I feel that articles about conditional prediction can be considered what you want, such as M2I

17. Are there any simulation projects suitable for performance testing of prediction algorithms that you can learn from and refer to?

A(stu): This paper is discussed: Choose Your Simulator Wisely A Review on Open-source Simulators for Autonomous Driving

18. How to estimate how much GPU memory is needed? If used For the Argoverse data set, how is it calculated? /mp.weixin.qq.com/s/EEkr8g4w0s2zhS_jmczUiA