import numpy as np
import pandas as pd
import os

os.chdir('D:\dataset\house_pries')

之前介绍了如何用pandas读取文件，今天就来详细的讲讲pandas和他的datafram结构

首先介绍如何自己建立个datafram框架

pd.DataFrame({'Yes':[50,21],'No':[131,2]})

pd.DataFrame({'Bob':['I liked it.','it is awful.'],'Sue':['Pretty good.','Bland.']})

一般来说，dataframe的行标签都是从0开始，正如上面所示那样，但是，有的时候我们想自己分配标签,这时我们就可以使用index这个指标来建构

pd.DataFrame({'Bob':['I liked it.','I was awful.'],
             'Sue':['Pretty good','Bland.']},
            index = ['product A','Product B'])

这里介绍一个概念 Series 。如果说dataframe是一个表格，那么Series就是其中的一列,唯一不同的是series

创建的方式很简单

pd.Series([1,2,3,4,5])

0    1
1    2
2    3
3    4
4    5
dtype: int64

因为Series本身就可以堪称dataframe的一列，那么series也可以自己分配行标签

pd.Series([30,35,40],index = ['S','E','R'])

S    30
E    35
R    40
dtype: int64

dataframe 与series 的关系是十分密切的，可以把dataframe看成是一系列series组成

pd.Series([30, 35, 40], index=['2015 Sales', '2016 Sales', '2017 Sales'], name='Product A')

2015 Sales    30
2016 Sales    35
2017 Sales    40
Name: Product A, dtype: int64

review = pd.read_csv("train.csv")

review.head()

如果你了解字典的话，你会知道字典的取值方式是用[]来取，pandas 也是这样

review['LotFrontage']

0        65.0
1        80.0
2        68.0
3        60.0
4        84.0
5        85.0
6        75.0
7         NaN
8        51.0
9        50.0
10       70.0
11       85.0
12        NaN
13       91.0
14        NaN
15       51.0
16        NaN
17       72.0
18       66.0
19       70.0
20      101.0
21       57.0
22       75.0
23       44.0
24        NaN
25      110.0
26       60.0
27       98.0
28       47.0
29       60.0
        ...  
1430     60.0
1431      NaN
1432     60.0
1433     93.0
1434     80.0
1435     80.0
1436     60.0
1437     96.0
1438     90.0
1439     80.0
1440     79.0
1441      NaN
1442     85.0
1443      NaN
1444     63.0
1445     70.0
1446      NaN
1447     80.0
1448     70.0
1449     21.0
1450     60.0
1451     78.0
1452     35.0
1453     90.0
1454     62.0
1455     62.0
1456     85.0
1457     66.0
1458     68.0
1459     75.0
Name: LotFrontage, Length: 1460, dtype: float64

当然你也可以使用.的方式来读取

review.LotFrontage

0        65.0
1        80.0
2        68.0
3        60.0
4        84.0
5        85.0
6        75.0
7         NaN
8        51.0
9        50.0
10       70.0
11       85.0
12        NaN
13       91.0
14        NaN
15       51.0
16        NaN
17       72.0
18       66.0
19       70.0
20      101.0
21       57.0
22       75.0
23       44.0
24        NaN
25      110.0
26       60.0
27       98.0
28       47.0
29       60.0
        ...  
1430     60.0
1431      NaN
1432     60.0
1433     93.0
1434     80.0
1435     80.0
1436     60.0
1437     96.0
1438     90.0
1439     80.0
1440     79.0
1441      NaN
1442     85.0
1443      NaN
1444     63.0
1445     70.0
1446      NaN
1447     80.0
1448     70.0
1449     21.0
1450     60.0
1451     78.0
1452     35.0
1453     90.0
1454     62.0
1455     62.0
1456     85.0
1457     66.0
1458     68.0
1459     75.0
Name: LotFrontage, Length: 1460, dtype: float64

那么两者之间有什么差别？

两者之间最大的不同是对于保留字符的处理上，比如当前的review有一个列，列名是Lot Front 那么这时[]操作就可以取这一列，而.就不行

review['LotFrontage'][0]

65.0

接下来就要讲索引

学会利用索引和下标操作是十分重要的，因为这两个操作在数据科学中的地位相当于空气和水

第一个是iloc

review.iloc[0]  #帮助你取数据的第一行

Id                     1
MSSubClass            60
MSZoning              RL
LotFrontage           65
LotArea             8450
Street              Pave
Alley                NaN
LotShape             Reg
LandContour          Lvl
Utilities         AllPub
LotConfig         Inside
LandSlope            Gtl
Neighborhood     CollgCr
Condition1          Norm
Condition2          Norm
BldgType            1Fam
HouseStyle        2Story
OverallQual            7
OverallCond            5
YearBuilt           2003
YearRemodAdd        2003
RoofStyle          Gable
RoofMatl         CompShg
Exterior1st      VinylSd
Exterior2nd      VinylSd
MasVnrType       BrkFace
MasVnrArea           196
ExterQual             Gd
ExterCond             TA
Foundation         PConc
                  ...   
BedroomAbvGr           3
KitchenAbvGr           1
KitchenQual           Gd
TotRmsAbvGrd           8
Functional           Typ
Fireplaces             0
FireplaceQu          NaN
GarageType        Attchd
GarageYrBlt         2003
GarageFinish         RFn
GarageCars             2
GarageArea           548
GarageQual            TA
GarageCond            TA
PavedDrive             Y
WoodDeckSF             0
OpenPorchSF           61
EnclosedPorch          0
3SsnPorch              0
ScreenPorch            0
PoolArea               0
PoolQC               NaN
Fence                NaN
MiscFeature          NaN
MiscVal                0
MoSold                 2
YrSold              2008
SaleType              WD
SaleCondition     Normal
SalePrice         208500
Name: 0, Length: 81, dtype: object

不管是loc 还是iloc 对于索引都是行优先，其次是列。但是在python的基本操作中则是列优先，其次是行。

这也就意味着对于这两个操作来说，行比列操作起来稍微容易一点点。

review.iloc[:,0]   #这种方式可以帮助你取第一列的值

0          1
1          2
2          3
3          4
4          5
5          6
6          7
7          8
8          9
9         10
10        11
11        12
12        13
13        14
14        15
15        16
16        17
17        18
18        19
19        20
20        21
21        22
22        23
23        24
24        25
25        26
26        27
27        28
28        29
29        30
        ... 
1430    1431
1431    1432
1432    1433
1433    1434
1434    1435
1435    1436
1436    1437
1437    1438
1438    1439
1439    1440
1440    1441
1441    1442
1442    1443
1443    1444
1444    1445
1445    1446
1446    1447
1447    1448
1448    1449
1449    1450
1450    1451
1451    1452
1452    1453
1453    1454
1454    1455
1455    1456
1456    1457
1457    1458
1458    1459
1459    1460
Name: Id, Length: 1460, dtype: int64

这里的:操作来自与python 不是pandas的基础语法，意思是所有事物。可以再跟其他操作结合，它也可以选取某一范围的列值。

review.iloc[:3,0]

0    1
1    2
2    3
Name: Id, dtype: int64

review.iloc[1:3,0]

1    2
2    3
Name: Id, dtype: int64

当然，你也可以通过列表的形式读取

review.iloc[[0,1,2],0]

0    1
1    2
2    3
Name: Id, dtype: int64

最后，你还需要知道python跟c/c++最明显的区别是下标的问题，python支持负数取数

review.iloc[-5:]  # 意思是从数据的末尾开始取五行的值

下面介绍一种基于标签的选择loc,这种方式选择的方式不是数据的位置，而是数据列或者行的名字。

review.loc[0,'LotFrontage']

65.0

因此直观上理解，iloc会比loc更加简单点，因为iloc是把数据当成一个矩阵来处理。但是另一方面来说，有的时候你知道某一列(行)的名字，却不知道具体位置，这时候loc的作用就体现出来了。

那么，在具体使用时我们应该怎么选取是应该使用iloc还是loc？这里提供一个小细节来帮助你选取

对于0:10 iloc会得到0,....9 但是loc则会得到0,.....10.

review.set_index("LotFrontage")  #这样可以使得你从另外一种角度来看数据

接下来，学习一些更好玩的操作。

review.LotFrontage == 65  #这种操作将会产生一个[T,F]series。
#当然这个操作也可以跟loc融合

0        True
1       False
2       False
3       False
4       False
5       False
6       False
7       False
8       False
9       False
10      False
11      False
12      False
13      False
14      False
15      False
16      False
17      False
18      False
19      False
20      False
21      False
22      False
23      False
24      False
25      False
26      False
27      False
28      False
29      False
        ...  
1430    False
1431    False
1432    False
1433    False
1434    False
1435    False
1436    False
1437    False
1438    False
1439    False
1440    False
1441    False
1442    False
1443    False
1444    False
1445    False
1446    False
1447    False
1448    False
1449    False
1450    False
1451    False
1452    False
1453    False
1454    False
1455    False
1456    False
1457    False
1458    False
1459    False
Name: LotFrontage, Length: 1460, dtype: bool

review.loc[review.LotFrontage==65]

这里我们还可以使用 & 😏

review.loc[(review.LotFrontage==65)&(review.LotArea >=6040)]

| 这个操作就交给你了

pandas 也有几个自带的选择器，这里介绍两个第一个是isin

review.loc[review.LotFrontage.isin([65,68])]

另一个则是isnull() ,这个可以帮助你把数据中的空值也就是nan显示出来

review.loc[review.Fence.notnull()]

最后，介绍一个分配值的方式

review['Id']= 666

review

	Id	MSSubClass	MSZoning	LotFrontage	LotArea	Street	Alley	LotShape	LandContour	Utilities	...	PoolQC	Fence	MiscFeature	MoSold	YrSold	SaleType	SaleCondition	SalePrice
0	1	60	RL	65.0	8450	Pave	NaN	Reg	Lvl	AllPub	...	NaN	NaN	NaN	2	2008	WD	Normal	208500
1	2	20	RL	80.0	9600	Pave	NaN	Reg	Lvl	AllPub	...	NaN	NaN	NaN	5	2007	WD	Normal	181500
2	3	60	RL	68.0	11250	Pave	NaN	IR1	Lvl	AllPub	...	NaN	NaN	NaN	9	2008	WD	Normal	223500
3	4	70	RL	60.0	9550	Pave	NaN	IR1	Lvl	AllPub	...	NaN	NaN	NaN	2	2006	WD	Abnorml	140000
4	5	60	RL	84.0	14260	Pave	NaN	IR1	Lvl	AllPub	...	NaN	NaN	NaN	12	2008	WD	Normal	250000

	Id	MSSubClass	MSZoning	LotFrontage	LotArea	Street	Alley	LotShape	LandContour	Utilities	...	PoolQC	Fence	MiscFeature	MiscVal	MoSold	YrSold	SaleType	SaleCondition	SalePrice
1455	1456	60	RL	62.0	7917	Pave	NaN	Reg	Lvl	AllPub	...	NaN	NaN	NaN	0	8	2007	WD	Normal	175000
1456	1457	20	RL	85.0	13175	Pave	NaN	Reg	Lvl	AllPub	...	NaN	MnPrv	NaN	0	2	2010	WD	Normal	210000
1457	1458	70	RL	66.0	9042	Pave	NaN	Reg	Lvl	AllPub	...	NaN	GdPrv	Shed	2500	5	2010	WD	Normal	266500
1458	1459	20	RL	68.0	9717	Pave	NaN	Reg	Lvl	AllPub	...	NaN	NaN	NaN	0	4	2010	WD	Normal	142125
1459	1460	20	RL	75.0	9937	Pave	NaN	Reg	Lvl	AllPub	...	NaN	NaN	NaN	0	6	2008	WD	Normal	147500

	Id	MSSubClass	MSZoning	LotArea	Street	Alley	LotShape	LandContour	Utilities	LotConfig	...	PoolArea	PoolQC	Fence	MiscFeature	MiscVal	MoSold	YrSold	SaleType	SaleCondition	SalePrice
LotFrontage
65.0	1	60	RL	8450	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	2	2008	WD	Normal	208500
80.0	2	20	RL	9600	Pave	NaN	Reg	Lvl	AllPub	FR2	...	0	NaN	NaN	NaN	0	5	2007	WD	Normal	181500
68.0	3	60	RL	11250	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	9	2008	WD	Normal	223500
60.0	4	70	RL	9550	Pave	NaN	IR1	Lvl	AllPub	Corner	...	0	NaN	NaN	NaN	0	2	2006	WD	Abnorml	140000
84.0	5	60	RL	14260	Pave	NaN	IR1	Lvl	AllPub	FR2	...	0	NaN	NaN	NaN	0	12	2008	WD	Normal	250000
85.0	6	50	RL	14115	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	MnPrv	Shed	700	10	2009	WD	Normal	143000
75.0	7	20	RL	10084	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	8	2007	WD	Normal	307000
NaN	8	60	RL	10382	Pave	NaN	IR1	Lvl	AllPub	Corner	...	0	NaN	NaN	Shed	350	11	2009	WD	Normal	200000
51.0	9	50	RM	6120	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	4	2008	WD	Abnorml	129900
50.0	10	190	RL	7420	Pave	NaN	Reg	Lvl	AllPub	Corner	...	0	NaN	NaN	NaN	0	1	2008	WD	Normal	118000
70.0	11	20	RL	11200	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	2	2008	WD	Normal	129500
85.0	12	60	RL	11924	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	7	2006	New	Partial	345000
NaN	13	20	RL	12968	Pave	NaN	IR2	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	9	2008	WD	Normal	144000
91.0	14	20	RL	10652	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	8	2007	New	Partial	279500
NaN	15	20	RL	10920	Pave	NaN	IR1	Lvl	AllPub	Corner	...	0	NaN	GdWo	NaN	0	5	2008	WD	Normal	157000
51.0	16	45	RM	6120	Pave	NaN	Reg	Lvl	AllPub	Corner	...	0	NaN	GdPrv	NaN	0	7	2007	WD	Normal	132000
NaN	17	20	RL	11241	Pave	NaN	IR1	Lvl	AllPub	CulDSac	...	0	NaN	NaN	Shed	700	3	2010	WD	Normal	149000
72.0	18	90	RL	10791	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	Shed	500	10	2006	WD	Normal	90000
66.0	19	20	RL	13695	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	6	2008	WD	Normal	159000
70.0	20	20	RL	7560	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	MnPrv	NaN	0	5	2009	COD	Abnorml	139000
101.0	21	60	RL	14215	Pave	NaN	IR1	Lvl	AllPub	Corner	...	0	NaN	NaN	NaN	0	11	2006	New	Partial	325300
57.0	22	45	RM	7449	Pave	Grvl	Reg	Bnk	AllPub	Inside	...	0	NaN	GdPrv	NaN	0	6	2007	WD	Normal	139400
75.0	23	20	RL	9742	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	9	2008	WD	Normal	230000
44.0	24	120	RM	4224	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	6	2007	WD	Normal	129900
NaN	25	20	RL	8246	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	MnPrv	NaN	0	5	2010	WD	Normal	154000
110.0	26	20	RL	14230	Pave	NaN	Reg	Lvl	AllPub	Corner	...	0	NaN	NaN	NaN	0	7	2009	WD	Normal	256300
60.0	27	20	RL	7200	Pave	NaN	Reg	Lvl	AllPub	Corner	...	0	NaN	NaN	NaN	0	5	2010	WD	Normal	134800
98.0	28	20	RL	11478	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2010	WD	Normal	306000
47.0	29	20	RL	16321	Pave	NaN	IR1	Lvl	AllPub	CulDSac	...	0	NaN	NaN	NaN	0	12	2006	WD	Normal	207500
60.0	30	30	RM	6324	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2008	WD	Normal	68500
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
60.0	1431	60	RL	21930	Pave	NaN	IR3	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	7	2006	WD	Normal	192140
NaN	1432	120	RL	4928	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	10	2009	WD	Normal	143750
60.0	1433	30	RL	10800	Pave	Grvl	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	8	2007	WD	Normal	64500
93.0	1434	60	RL	10261	Pave	NaN	IR1	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2008	WD	Normal	186500
80.0	1435	20	RL	17400	Pave	NaN	Reg	Low	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2006	WD	Normal	160000
80.0	1436	20	RL	8400	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	GdPrv	NaN	0	7	2008	COD	Abnorml	174000
60.0	1437	20	RL	9000	Pave	NaN	Reg	Lvl	AllPub	FR2	...	0	NaN	GdWo	NaN	0	5	2007	WD	Normal	120500
96.0	1438	20	RL	12444	Pave	NaN	Reg	Lvl	AllPub	FR2	...	0	NaN	NaN	NaN	0	11	2008	New	Partial	394617
90.0	1439	20	RM	7407	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	MnPrv	NaN	0	4	2010	WD	Normal	149700
80.0	1440	60	RL	11584	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	11	2007	WD	Normal	197000
79.0	1441	70	RL	11526	Pave	NaN	IR1	Bnk	AllPub	Inside	...	0	NaN	NaN	NaN	0	9	2008	WD	Normal	191000
NaN	1442	120	RM	4426	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2008	WD	Normal	149300
85.0	1443	60	FV	11003	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	4	2009	WD	Normal	310000
NaN	1444	30	RL	8854	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2009	WD	Normal	121000
63.0	1445	20	RL	8500	Pave	NaN	Reg	Lvl	AllPub	FR2	...	0	NaN	NaN	NaN	0	11	2007	WD	Normal	179600
70.0	1446	85	RL	8400	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2007	WD	Normal	129000
NaN	1447	20	RL	26142	Pave	NaN	IR1	Lvl	AllPub	CulDSac	...	0	NaN	NaN	NaN	0	4	2010	WD	Normal	157900
80.0	1448	60	RL	10000	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	12	2007	WD	Normal	240000
70.0	1449	50	RL	11767	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	GdWo	NaN	0	5	2007	WD	Normal	112000
21.0	1450	180	RM	1533	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	8	2006	WD	Abnorml	92000
60.0	1451	90	RL	9000	Pave	NaN	Reg	Lvl	AllPub	FR2	...	0	NaN	NaN	NaN	0	9	2009	WD	Normal	136000
78.0	1452	20	RL	9262	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2009	New	Partial	287090
35.0	1453	180	RM	3675	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	5	2006	WD	Normal	145000
90.0	1454	20	RL	17217	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	7	2006	WD	Abnorml	84500
62.0	1455	20	FV	7500	Pave	Pave	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	10	2009	WD	Normal	185000
62.0	1456	60	RL	7917	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	8	2007	WD	Normal	175000
85.0	1457	20	RL	13175	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	MnPrv	NaN	0	2	2010	WD	Normal	210000
66.0	1458	70	RL	9042	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	GdPrv	Shed	2500	5	2010	WD	Normal	266500
68.0	1459	20	RL	9717	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	4	2010	WD	Normal	142125
75.0	1460	20	RL	9937	Pave	NaN	Reg	Lvl	AllPub	Inside	...	0	NaN	NaN	NaN	0	6	2008	WD	Normal	147500

	Yes	No
0	50	131
1	21	2

	Bob	Sue
0	I liked it.	Pretty good.
1	it is awful.	Bland.

	Bob	Sue
product A	I liked it.	Pretty good
Product B	I was awful.	Bland.