爬取B站热门视频数据并对数据进行可视化呈现和可视化数据分析

qq1501036850zsr / 2023-06-05 / 原文

(一)选题背景

随着网络的飞速发展,各各视频平台也在不断兴起,每一个视频平台也各具自己的特色,有以短视频为主的抖音,也有以二次元文化为主的B站,以搞笑视频为主的快手,它们都各具自己的特色,视频的大量发布,面向不同的人群,质量也参差不齐,经过大浪淘沙,我们可以看到一些播放量高,评论量多的优秀视频,那么我们就可以在平台上找到播放量高,高质量视频并把它们统计分析进行可视话呈现,方便人们选择视频.

(二)步骤解释

1.爬取视频数据

找到数据地址

对爬取到的数据进行解析，获取每一条视频数据信息

对数据进行保存

2.对数据进行可视化

使用flask框架，
挑选合适的Echarts图表模板
把爬取到的数据通过flask框架传进Echart模板中

3,可视话分析

数据分析

爬取网站地址:https://api.bilibili.com/x/web-interface/popular?ps=20&pn=1

(三)具体步骤

1.找到数据地址

2.对爬取到的数据进行解析，获取每一条视频数据信息

import requests
import json
import csv
all_data=[]
def Get_data():
    header={
        'user-agent':'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36 Edg/114.0.1823.37'
    }
    url='https://api.bilibili.com/x/web-interface/popular'
    for i in range(1,30):
        param={
            'ps': 20,
            'pn': i
        }
        res=requests.get(url=url,headers=header,params=param)
        #print(res.text)
        Parse(res.text)
        res.close()

3.对数据解析

def Parse(text):
    dict=json.loads(text)
    # print(dict['data']['list'])
    for i in dict['data']['list']:
        # print(i['title'])
        # print(i['owner']['name'])
        # print(i['stat']['view'])
        # print(i['stat']['danmaku'])
        temp=(i['title'],
              i['owner']['name'],
              i['stat']['view'],
              i['stat']['danmaku']
              )
        all_data.append(temp)

4.对数据进行保存

def Save_data(all_data):
    with open(r'all_data1.csv',mode='w',encoding='utf-8',newline='') as fp:
        writer=csv.writer(fp)
        writer.writerow(['视频名称',
                         '视频作者',
                         '视频播放量',
                         '视频评论数']
                        )
        for i in all_data:
            writer.writerow(i)

if __name__ == '__main__':
    #Get_data()
    #print(all_data)
    #Save_data(all_data=all_data)
    with open('all_data.csv',mode='r',encoding='utf-8') as fp:
        reader=csv.DictReader(fp)
        for i in reader:
           print(i)

5.对数据进行列表可视话

@app.route('/1')
def demo():
    data=[]
    with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv',mode='r',encoding='utf-8') as fp:
        reader=csv.reader(fp)
        for i in reader:
            data.append(i)
    return render_template('1.html',alldata=data)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>1</title>
    <link rel="stylesheet" href="https://cdn.staticfile.org/twitter-bootstrap/4.3.1/css/bootstrap.min.css">
    <script src="https://cdn.staticfile.org/jquery/3.2.1/jquery.min.js"></script>
    <script src="https://cdn.staticfile.org/popper.js/1.15.0/umd/popper.min.js"></script>
    <script src="https://cdn.staticfile.org/twitter-bootstrap/4.3.1/js/bootstrap.min.js"></script>
</head>
<body>
    <table class="table table-striped table-bordered table-hover">
        {%for i in alldata%}
        <tr>
            <td>{{ i[0] }}</td>
            <td>{{ i[1] }}</td>
            <td>{{ i[2] }}</td>
            <td>{{ i[3] }}</td>
        </tr>
        {%endfor%}
    </table>
</body>
</html>

6.对数据进行图表可视化

@app.route('/2')
def demodemo():
    title = []
    up=[]
    view=[]
    comment=[]
    with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv', mode='r', encoding='utf-8') as fp:
        reader = csv.DictReader(fp)
        for i in reader:
            title.append(i['视频名称'])
            up.append(i['up主'])
            view.append(i['播放量'])
            comment.append(i['评论数'])
    return render_template('2.html',title=title,view=view,comment=comment,up=up)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>2</title>
    <script src="https://cdn.staticfile.org/echarts/4.3.0/echarts.min.js"></script>
</head>
<body>
    <div id="main" style="width: 100%;height:800px;"></div>
    <script type="text/javascript">
        var myChart = echarts.init(document.getElementById('main'));
        option = {
            tooltip: {
                trigger: "axis",
                axisPointer: {
                  type: "shadow"
                }
            },
            dataZoom: [
                {
                    type: 'slider',
                    show: true,
                    start: 94,
                    end: 100,
                    handleSize: 8
                },
                {
                    type: 'inside',
                    start: 94,
                    end: 100
                },
                {
                    type: 'slider',
                    show: true,
                    yAxisIndex: 0,
                    filterMode: 'empty',
                    width: 12,
                    height: '70%',
                    handleSize: 8,
                    showDataShadow: false,
                    left: '93%'
                }
            ],
            xAxis: {
                type: 'category',
                data: {{ title|tojson }},
                axisLabel:{
                    interval:0,
                    rotate:60
                }
            },
            yAxis: {
                type: 'value'
            },
            series: [{
                data: {{ view|tojson }},
                type: 'bar',
                showBackground: true,
                backgroundStyle: {
                    color: 'rgba(180, 180, 180, 0.2)'
                }
            }]
        };
        
        myChart.setOption(option);
    </script>
</body>
</html>

7.可视化分析

@app.route('/3')
def echart():
    title=[]
    view=[]
    comment=[]
    with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv', mode='r', encoding='utf-8') as fp:
        reader = csv.DictReader(fp)
        for i in reader:
            title.append(i['视频名称'])
            view.append(i['播放量'])
            comment.append(i['评论数'])
    return render_template('3.html',title=title,view=view,comment=comment)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>可视化分析</title>
    <script src="https://cdn.staticfile.org/echarts/4.3.0/echarts.min.js"></script>
</head>
<body>
    <div id="main" style="width: 100%;height:800px;"></div>
    <div id="main1" style="width: 100%;height:800px;"></div>
    <script type="text/javascript">
        var myChart = echarts.init(document.getElementById('main'));
        var option = {
            title: {
                text: '播放量和评论数可视化'
            },
            dataZoom: [
                {
                    type: 'slider',
                    show: true,
                    start: 94,
                    end: 100,
                    handleSize: 8
                },
                {
                    type: 'inside',
                    start: 94,
                    end: 100
                },
                {
                    type: 'slider',
                    show: true,
                    yAxisIndex: 0,
                    filterMode: 'empty',
                    width: 12,
                    height: '70%',
                    handleSize: 8,
                    showDataShadow: false,
                    left: '93%'
                }
            ],
            grid:{

            },
            tooltip: {
                trigger: "axis",
                axisPointer: {
                  type: "shadow"
                }
            },
            legend: {
                data:['视频播放量','视频评论数']
            },
            xAxis: [{
                type: "category",
                data: {{ title| tojson }},
                axisLabel: {
                  interval: 0,
                  show: true,
                  formatter: function(value) {
                      return value.split('').join('\n')
                    }
                }
            }],
            yAxis: {
                type:'value',

            },
            series: [{
                label: {
                    show: true,
                    position: 'inside'
                },
                name: '视频播放量',
                type: 'bar',
                data: {{ view|tojson }}
            },{
                label: {
                    show: true,
                    position: 'inside'
                },
                name: '视频评论数',
                type: 'bar',
                data: {{ comment|tojson }}
            }]
        };

        myChart.setOption(option);
    </script>
    <script type="text/javascript">
        var myChart = echarts.init(document.getElementById('main1'));

        var option = {
            title: {
                text: '评论数可视化'
            },
            dataZoom: [
                {
                    type: 'slider',
                    show: true,
                    start: 94,
                    end: 100,
                    handleSize: 8
                },
                {
                    type: 'inside',
                    start: 94,
                    end: 100
                },
                {
                    type: 'slider',
                    show: true,
                    yAxisIndex: 0,
                    filterMode: 'empty',
                    width: 12,
                    height: '70%',
                    handleSize: 8,
                    showDataShadow: false,
                    left: '93%'
                }
            ],
            tooltip: {},
            legend: {
                data:['视频评论数']
            },
            xAxis: [{
                type: "category",
                data: {{ title| tojson }},
                axisLabel: {
                  interval: 0,
                  show: true,
                  formatter: function(value) {
                      return value.split('').join('\n')
                    }
                }
            }],
            yAxis: {},
            series: [{
                name: '视频评论数',
                type: 'bar',
                data: {{ comment|tojson }}
            }
            ]
        };

        myChart.setOption(option);
    </script>
</body>
</html>

(四)总代码

  1 import requests
  2 import json
  3 import csv
  4 all_data=[]
  5 def Get_data():
  6     header={
  7         'user-agent':'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36 Edg/114.0.1823.37'
  8     }
  9     url='https://api.bilibili.com/x/web-interface/popular'
 10     for i in range(1,30):
 11         param={
 12             'ps': 20,
 13             'pn': i
 14         }
 15         res=requests.get(url=url,headers=header,params=param)
 16         #print(res.text)
 17         Parse(res.text)
 18         res.close()
 19 def Parse(text):
 20     dict=json.loads(text)
 21     # print(dict['data']['list'])
 22     for i in dict['data']['list']:
 23         # print(i['title'])
 24         # print(i['owner']['name'])
 25         # print(i['stat']['view'])
 26         # print(i['stat']['danmaku'])
 27         temp=(i['title'],
 28               i['owner']['name'],
 29               i['stat']['view'],
 30               i['stat']['danmaku']
 31               )
 32         all_data.append(temp)
 33 def Save_data(all_data):
 34     with open(r'all_data1.csv',mode='w',encoding='utf-8',newline='') as fp:
 35         writer=csv.writer(fp)
 36         writer.writerow(['视频名称',
 37                          '视频作者',
 38                          '视频播放量',
 39                          '视频评论数']
 40                         )
 41         for i in all_data:
 42             writer.writerow(i)
 43 
 44 if __name__ == '__main__':
 45     #Get_data()
 46     #print(all_data)
 47     #Save_data(all_data=all_data)
 48     with open('all_data.csv',mode='r',encoding='utf-8') as fp:
 49         reader=csv.DictReader(fp)
 50         for i in reader:
 51            print(i)
 52 
 53 
 54 from flask import Flask,render_template
 55 import csv
 56 
 57 app=Flask(__name__)
 58 
 59 @app.route('/')
 60 def hello_world():
 61 
 62     return '你好世界'
 63 @app.route('/data')
 64 def data():
 65     data = []
 66     with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv', mode='r', encoding='utf-8') as fp:
 67         reader = csv.reader(fp)
 68         for i in reader:
 69             if len(i) != 0:
 70                 data.append(i)
 71     return render_template('data.html',data=data)
 72 
 73 @app.route('/3')
 74 def echart():
 75     title=[]
 76     view=[]
 77     comment=[]
 78     with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv', mode='r', encoding='utf-8') as fp:
 79         reader = csv.DictReader(fp)
 80         for i in reader:
 81             title.append(i['视频名称'])
 82             view.append(i['播放量'])
 83             comment.append(i['评论数'])
 84     return render_template('3.html',title=title,view=view,comment=comment)
 85 
 86 @app.route('/1')
 87 def demo():
 88     data=[]
 89     with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv',mode='r',encoding='utf-8') as fp:
 90         reader=csv.reader(fp)
 91         for i in reader:
 92             data.append(i)
 93     return render_template('1.html',alldata=data)
 94 
 95 @app.route('/2')
 96 def demodemo():
 97     title = []
 98     up=[]
 99     view=[]
100     comment=[]
101     with open(r'C:\Users\chase\PycharmProjects\pythonProject\flask666\all_data.csv', mode='r', encoding='utf-8') as fp:
102         reader = csv.DictReader(fp)
103         for i in reader:
104             title.append(i['视频名称'])
105             up.append(i['up主'])
106             view.append(i['播放量'])
107             comment.append(i['评论数'])
108     return render_template('2.html',title=title,view=view,comment=comment,up=up)
109 if __name__=='__main__':
110     app.run(debug=True)
111 
112 
113 data.html
114 
115 <!DOCTYPE html>
116 <html lang="en">
117 <head>
118     <meta charset="UTF-8">
119     <title>数据分析</title>
120     <!-- 新 Bootstrap4 核心 CSS 文件 -->
121     <link rel="stylesheet" href="https://cdn.staticfile.org/twitter-bootstrap/4.3.1/css/bootstrap.min.css">
122 
123     <!-- jQuery文件。务必在bootstrap.min.js 之前引入 -->
124     <script src="https://cdn.staticfile.org/jquery/3.2.1/jquery.min.js"></script>
125 
126     <!-- bootstrap.bundle.min.js 用于弹窗、提示、下拉菜单，包含了 popper.min.js -->
127     <script src="https://cdn.staticfile.org/popper.js/1.15.0/umd/popper.min.js"></script>
128 
129     <!-- 最新的 Bootstrap4 核心 JavaScript 文件 -->
130     <script src="https://cdn.staticfile.org/twitter-bootstrap/4.3.1/js/bootstrap.min.js"></script>
131 
132 
133 </head>
134 <body>
135     <table class="table table-striped table-bordered table-hover">
136         {% for i in data %}
137         <tr>
138             <td>{{ i[0] }}</td>
139             <td>{{ i[1] }}</td>
140             <td>{{ i[2] }}</td>
141             <td>{{ i[3] }}</td>
142         </tr>
143         {%endfor%}
144     </table>
145 </body>
146 </html>
147 
148 
149 1.html
150 
151 <!DOCTYPE html>
152 <html lang="en">
153 <head>
154     <meta charset="UTF-8">
155     <title>1</title>
156     <link rel="stylesheet" href="https://cdn.staticfile.org/twitter-bootstrap/4.3.1/css/bootstrap.min.css">
157     <script src="https://cdn.staticfile.org/jquery/3.2.1/jquery.min.js"></script>
158     <script src="https://cdn.staticfile.org/popper.js/1.15.0/umd/popper.min.js"></script>
159     <script src="https://cdn.staticfile.org/twitter-bootstrap/4.3.1/js/bootstrap.min.js"></script>
160 </head>
161 <body>
162     <table class="table table-striped table-bordered table-hover">
163         {%for i in alldata%}
164         <tr>
165             <td>{{ i[0] }}</td>
166             <td>{{ i[1] }}</td>
167             <td>{{ i[2] }}</td>
168             <td>{{ i[3] }}</td>
169         </tr>
170         {%endfor%}
171     </table>
172 </body>
173 </html>
174 
175 2.html
176 
177 <!DOCTYPE html>
178 <html lang="en">
179 <head>
180     <meta charset="UTF-8">
181     <title>2</title>
182     <script src="https://cdn.staticfile.org/echarts/4.3.0/echarts.min.js"></script>
183 </head>
184 <body>
185     <div id="main" style="width: 100%;height:800px;"></div>
186     <script type="text/javascript">
187         var myChart = echarts.init(document.getElementById('main'));
188         option = {
189             tooltip: {
190                 trigger: "axis",
191                 axisPointer: {
192                   type: "shadow"
193                 }
194             },
195             dataZoom: [
196                 {
197                     type: 'slider',
198                     show: true,
199                     start: 94,
200                     end: 100,
201                     handleSize: 8
202                 },
203                 {
204                     type: 'inside',
205                     start: 94,
206                     end: 100
207                 },
208                 {
209                     type: 'slider',
210                     show: true,
211                     yAxisIndex: 0,
212                     filterMode: 'empty',
213                     width: 12,
214                     height: '70%',
215                     handleSize: 8,
216                     showDataShadow: false,
217                     left: '93%'
218                 }
219             ],
220             xAxis: {
221                 type: 'category',
222                 data: {{ title|tojson }},
223                 axisLabel:{
224                     interval:0,
225                     rotate:60
226                 }
227             },
228             yAxis: {
229                 type: 'value'
230             },
231             series: [{
232                 data: {{ view|tojson }},
233                 type: 'bar',
234                 showBackground: true,
235                 backgroundStyle: {
236                     color: 'rgba(180, 180, 180, 0.2)'
237                 }
238             }]
239         };
240 
241         myChart.setOption(option);
242     </script>
243 </body>
244 </html>
245 
246 3.html
247 
248 <!DOCTYPE html>
249 <html lang="en">
250 <head>
251     <meta charset="UTF-8">
252     <title>可视化分析</title>
253     <script src="https://cdn.staticfile.org/echarts/4.3.0/echarts.min.js"></script>
254 </head>
255 <body>
256     <div id="main" style="width: 100%;height:800px;"></div>
257     <div id="main1" style="width: 100%;height:800px;"></div>
258     <script type="text/javascript">
259         var myChart = echarts.init(document.getElementById('main'));
260         var option = {
261             title: {
262                 text: '播放量和评论数可视化'
263             },
264             dataZoom: [
265                 {
266                     type: 'slider',
267                     show: true,
268                     start: 94,
269                     end: 100,
270                     handleSize: 8
271                 },
272                 {
273                     type: 'inside',
274                     start: 94,
275                     end: 100
276                 },
277                 {
278                     type: 'slider',
279                     show: true,
280                     yAxisIndex: 0,
281                     filterMode: 'empty',
282                     width: 12,
283                     height: '70%',
284                     handleSize: 8,
285                     showDataShadow: false,
286                     left: '93%'
287                 }
288             ],
289             grid:{
290 
291             },
292             tooltip: {
293                 trigger: "axis",
294                 axisPointer: {
295                   type: "shadow"
296                 }
297             },
298             legend: {
299                 data:['视频播放量','视频评论数']
300             },
301             xAxis: [{
302                 type: "category",
303                 data: {{ title| tojson }},
304                 axisLabel: {
305                   interval: 0,
306                   show: true,
307                   formatter: function(value) {
308                       return value.split('').join('\n')
309                     }
310                 }
311             }],
312             yAxis: {
313                 type:'value',
314 
315             },
316             series: [{
317                 label: {
318                     show: true,
319                     position: 'inside'
320                 },
321                 name: '视频播放量',
322                 type: 'bar',
323                 data: {{ view|tojson }}
324             },{
325                 label: {
326                     show: true,
327                     position: 'inside'
328                 },
329                 name: '视频评论数',
330                 type: 'bar',
331                 data: {{ comment|tojson }}
332             }]
333         };
334 
335         myChart.setOption(option);
336     </script>
337     <script type="text/javascript">
338         var myChart = echarts.init(document.getElementById('main1'));
339 
340         var option = {
341             title: {
342                 text: '评论数可视化'
343             },
344             dataZoom: [
345                 {
346                     type: 'slider',
347                     show: true,
348                     start: 94,
349                     end: 100,
350                     handleSize: 8
351                 },
352                 {
353                     type: 'inside',
354                     start: 94,
355                     end: 100
356                 },
357                 {
358                     type: 'slider',
359                     show: true,
360                     yAxisIndex: 0,
361                     filterMode: 'empty',
362                     width: 12,
363                     height: '70%',
364                     handleSize: 8,
365                     showDataShadow: false,
366                     left: '93%'
367                 }
368             ],
369             tooltip: {},
370             legend: {
371                 data:['视频评论数']
372             },
373             xAxis: [{
374                 type: "category",
375                 data: {{ title| tojson }},
376                 axisLabel: {
377                   interval: 0,
378                   show: true,
379                   formatter: function(value) {
380                       return value.split('').join('\n')
381                     }
382                 }
383             }],
384             yAxis: {},
385             series: [{
386                 name: '视频评论数',
387                 type: 'bar',
388                 data: {{ comment|tojson }}
389             }
390             ]
391         };
392 
393         myChart.setOption(option);
394     </script>
395 </body>
396 </html>