熟悉pandas的pythoner 应该知道给dataframe增加一列很容易,直接以字典形式指定就好了,pyspark中就不同了,摸索了一下,可以使用如下方式增加
from pyspark import SparkContext from pyspark import SparkConf from pypsark.sql import SparkSession from pyspark.sql import functions spark = SparkSession.builder.config(conf=SparkConf()).getOrCreate() data = [['Alice', 19, 'blue', '["Alice", 19, "blue"]'], ['Jane', 20, 'green', '["Jane", 20, "green"]'], ['Mary', 21, 'blue', '["Mary", 21, "blue"]'], ] frame = spark.createDataFrame(data, schema=["name", "age", "eye_color", "detail"]) frame.cache() frame.show()
+-----+---+---------+--------------------+
| name|age|eye_color| detail|
+-----+---+---------+--------------------+
|Alice| 19| blue|["Alice", 19, "bl...|
| Jane| 20| green|["Jane", 20, "gre...|
| Mary| 21| blue|["Mary", 21, "blue"]|
+-----+---+---------+--------------------+
1、 增加常数项
frame2 = frame.withColumn("contant", functions.lit(10)) frame2.show()
+-----+---+---------+--------------------+-------+
| name|age|eye_color| detail|contant|
+-----+---+---------+--------------------+-------+
|Alice| 19| blue|["Alice", 19, "bl...| 10|
| Jane| 20| green|["Jane", 20, "gre...| 10|
| Mary| 21| blue|["Mary", 21, "blue"]| 10|
+-----+---+---------+--------------------+-------+
2、简单根据某列进行计算
2.1 使用 withColumn
frame3_1 = frame.withColumn("name_length", functions.length(frame.name)) frame3_1.show()
+-----+---+---------+--------------------+-----------+
| name|age|eye_color| detail|name_length|
+-----+---+---------+--------------------+-----------+
|Alice| 19| blue|["Alice", 19, "bl...| 5|
| Jane| 20| green|["Jane", 20, "gre...| 4|
| Mary| 21| blue|["Mary", 21, "blue"]| 4|
+-----+---+---------+--------------------+-----------+
2.2 使用 select
frame3_2 = frame.select(["name", functions.length(frame.name).alias("name_length")]) frame3_2.show()
+-----+-----------+
| name|name_length|
+-----+-----------+
|Alice| 5|
| Jane| 4|
| Mary| 4|
+-----+-----------+
2.3 使用 selectExpr
frame3_3 = frame.selectExpr(["name", "length(name) as name_length"]) frame3_3.show()
+-----+-----------+
| name|name_length|
+-----+-----------+
|Alice| 5|
| Jane| 4|
| Mary| 4|
+-----+-----------+
3、定制化根据某列进行计算
比如我想对某列做指定操作,但是对应的函数没得咋办,造,自己造~
frame4 = frame.withColumn("detail_length", functions.UserDefinedFunction(lambda obj: len(json.loads(obj)))(frame.detail)) # or def length_detail(obj): return len(json.loads(obj)) frame4 = frame.withColumn("detail_length", functions.UserDefinedFunction(length_detail)(frame.detail)) frame4.show()
+-----+---+---------+--------------------+-------------+
| name|age|eye_color| detail|detail_length|
+-----+---+---------+--------------------+-------------+
|Alice| 19| blue|["Alice", 19, "bl...| 3|
| Jane| 20| green|["Jane", 20, "gre...| 3|
| Mary| 21| blue|["Mary", 21, "blue"]| 3|
+-----+---+---------+--------------------+-------------+
RTX 5090要首发 性能要翻倍!三星展示GDDR7显存
三星在GTC上展示了专为下一代游戏GPU设计的GDDR7内存。
首次推出的GDDR7内存模块密度为16GB,每个模块容量为2GB。其速度预设为32 Gbps(PAM3),但也可以降至28 Gbps,以提高产量和初始阶段的整体性能和成本效益。
据三星表示,GDDR7内存的能效将提高20%,同时工作电压仅为1.1V,低于标准的1.2V。通过采用更新的封装材料和优化的电路设计,使得在高速运行时的发热量降低,GDDR7的热阻比GDDR6降低了70%。
更新日志
- 群星《前途海量 电影原声专辑》[FLAC/分轨][227.78MB]
- 张信哲.1992-知道新曲与精丫巨石】【WAV+CUE】
- 王翠玲.1995-ANGEL【新艺宝】【WAV+CUE】
- 景冈山.1996-我的眼里只有你【大地唱片】【WAV+CUE】
- 群星《八戒 电影原声带》[320K/MP3][188.97MB]
- 群星《我的阿勒泰 影视原声带》[320K/MP3][139.47MB]
- 纪钧瀚《胎教古典音乐 钢琴与大提琴的沉浸时光》[320K/MP3][148.91MB]
- 刘雅丽.2001-丽花皇后·EMI精选王【EMI百代】【FLAC分轨】
- 齐秦.1994-黄金十年1981-1990CHINA.TOUR.LIVE精丫上华】【WAV+CUE】
- 群星.2008-本色·百代音乐人创作专辑【EMI百代】【WAV+CUE】
- 群星.2001-同步过冬AVCD【环球】【WAV+CUE】
- 群星.2020-同步过冬2020冀待晴空【环球】【WAV+CUE】
- 沈雁.1986-四季(2012梦田复刻版)【白云唱片】【WAV+CUE】
- 纪钧瀚《胎教古典音乐 钢琴与大提琴的沉浸时光》[FLAC/分轨][257.88MB]
- 《国语老歌 怀旧篇 3CD》[WAV/分轨][1.6GB]